Nov 02 13:56:34.182672 ip-10-0-129-115 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:56:34.593097 ip-10-0-129-115 kubenswrapper[2542]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:56:34.593097 ip-10-0-129-115 kubenswrapper[2542]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:56:34.593097 ip-10-0-129-115 kubenswrapper[2542]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:56:34.593097 ip-10-0-129-115 kubenswrapper[2542]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.597910    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598011    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598017    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598021    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598026    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598031    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598044    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598048    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598053    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598058    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598063    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598067    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598071    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598075    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598079    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598083    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598088    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598092    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.598510 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598097    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598106    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598112    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598116    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598120    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598124    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598128    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598132    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598136    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598140    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598144    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598148    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598152    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598157    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598166    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598171    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598176    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598181    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598185    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598189    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.599048 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598193    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598197    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598201    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598205    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598209    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598213    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598222    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598226    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598231    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598235    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598240    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598244    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598248    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598252    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598259    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598264    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598268    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598272    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598281    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598285    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.599543 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598289    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598294    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598298    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598302    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598306    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598310    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598315    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598320    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598324    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598328    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598332    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598341    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598345    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598349    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598373    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598377    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598381    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598386    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598392    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.600011 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598399    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598404    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598408    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598413    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598422    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598427    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598431    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598435    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598440    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598445    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598449    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598453    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598458    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598462    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598466    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598471    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598489    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598513    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598519    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.600478 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598524    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598553    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598559    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598702    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598707    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598718    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598724    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598730    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598734    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.598738    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599668    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599677    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599680    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599683    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599686    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599689    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599692    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599695    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599698    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599700    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.600914 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599703    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599705    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599708    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599710    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599713    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599715    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599718    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599720    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599723    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599726    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599740    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599744    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599747    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599750    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599753    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599757    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599759    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599762    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599765    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599768    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.601455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599770    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599773    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599775    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599778    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599780    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599783    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599786    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599788    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599791    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599793    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599796    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599798    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599801    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599803    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599806    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599808    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599811    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599813    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599815    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.601970 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599818    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599820    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599823    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599826    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599830    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599832    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599835    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599837    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599841    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599843    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599846    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599848    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599851    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599853    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599857    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599861    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599865    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599868    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599870    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.602436 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599873    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599875    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599878    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599880    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599882    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599885    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599887    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599890    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599898    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599901    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599903    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599906    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599908    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599910    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599913    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599917    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599920    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599922    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599925    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599927    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.602883 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599930    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599933    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599935    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599938    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599940    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599942    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599946    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599950    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599953    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599957    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599960    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599962    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599965    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599967    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599970    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599973    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599975    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.599978    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601325    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.603417 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601334    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601344    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601348    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601370    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601374    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601377    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601379    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601382    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601385    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601388    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601391    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601394    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601396    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601399    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601401    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601404    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601407    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601409    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601412    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.603882 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601414    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601417    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601419    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601422    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601424    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601427    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601429    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601432    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601434    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601436    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601439    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601441    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601444    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601447    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601450    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601458    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601461    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601464    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601467    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.604323 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601470    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601473    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601475    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601477    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601481    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601485    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601488    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601490    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601493    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601495    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601498    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601500    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601503    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601505    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601508    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601510    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601512    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601515    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601517    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.604784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601520    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601522    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601526    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601529    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601531    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601534    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601536    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601539    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601542    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601544    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601553    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601556    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601558    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601561    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601564    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601566    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601569    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601571    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601574    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601576    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.605233 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601579    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601581    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601584    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601587    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601589    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601592    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601594    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601596    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601599    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601602    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601604    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601607    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601609    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601611    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601614    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601616    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601619    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601621    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601624    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601627    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.605722 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601629    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601633    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601637    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601646    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601651    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601654    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601657    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.601660    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601775    2542 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601786    2542 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601804    2542 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601810    2542 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601814    2542 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601817    2542 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601824    2542 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601828    2542 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601831    2542 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601834    2542 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601837    2542 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601840    2542 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601843    2542 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601846    2542 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:56:34.606184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601849    2542 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601852    2542 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601855    2542 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601857    2542 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601865    2542 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601867    2542 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601870    2542 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601873    2542 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601877    2542 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601881    2542 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601884    2542 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601888    2542 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601891    2542 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601894    2542 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601902    2542 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601906    2542 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601909    2542 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601912    2542 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601916    2542 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601919    2542 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601922    2542 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601925    2542 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601928    2542 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601931    2542 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601939    2542 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:56:34.606728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601942    2542 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601945    2542 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601948    2542 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601951    2542 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601955    2542 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601958    2542 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601961    2542 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601964    2542 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601967    2542 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601970    2542 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601974    2542 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601977    2542 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601980    2542 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601983    2542 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601985    2542 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601989    2542 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601995    2542 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.601998    2542 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602001    2542 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602004    2542 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602007    2542 flags.go:64] FLAG: --help="false"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602010    2542 flags.go:64] FLAG: --hostname-override="ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602019    2542 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602022    2542 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:56:34.607318 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602025    2542 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602029    2542 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602032    2542 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602035    2542 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602037    2542 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602040    2542 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602043    2542 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602046    2542 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602049    2542 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602051    2542 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602054    2542 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602057    2542 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602060    2542 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602063    2542 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602065    2542 flags.go:64] FLAG: --lock-file=""
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602069    2542 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602072    2542 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602075    2542 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602080    2542 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602083    2542 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602086    2542 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602089    2542 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602091    2542 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602095    2542 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:56:34.607915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602099    2542 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602102    2542 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602106    2542 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602109    2542 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602113    2542 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602116    2542 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602119    2542 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602127    2542 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602130    2542 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602132    2542 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602135    2542 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602138    2542 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602145    2542 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602148    2542 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602151    2542 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602154    2542 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602157    2542 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602160    2542 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602163    2542 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602165    2542 flags.go:64] FLAG: --port="10250"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602168    2542 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602171    2542 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a897f8dc18ece89"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602174    2542 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602177    2542 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602180    2542 flags.go:64] FLAG: --register-node="true"
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602183    2542 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:56:34.608544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602187    2542 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602189    2542 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602192    2542 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602195    2542 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602199    2542 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602202    2542 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602207    2542 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602210    2542 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602213    2542 flags.go:64] FLAG: --runonce="false"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602216    2542 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602219    2542 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602221    2542 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602224    2542 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602227    2542 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602236    2542 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602239    2542 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602242    2542 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602244    2542 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602247    2542 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602250    2542 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602253    2542 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602256    2542 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602258    2542 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602261    2542 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602266    2542 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:56:34.609287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602269    2542 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602272    2542 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602278    2542 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602281    2542 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602284    2542 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602287    2542 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602290    2542 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602292    2542 flags.go:64] FLAG: --v="2"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602297    2542 flags.go:64] FLAG: --version="false"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602301    2542 flags.go:64] FLAG: --vmodule=""
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602305    2542 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602308    2542 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602470    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602477    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602480    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602483    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602486    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602488    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602491    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602494    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602496    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602499    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.609901 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602508    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602511    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602515    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602519    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602522    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602525    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602528    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602531    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602533    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602536    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602538    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602541    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602543    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602546    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602548    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602550    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602553    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602555    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602558    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602560    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.611207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602563    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602565    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602568    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602571    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602574    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602577    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602579    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602582    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602584    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602587    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602589    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602592    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602594    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602602    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602605    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602608    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602610    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602613    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602615    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.611784 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602617    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602620    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602622    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602625    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602628    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602630    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602633    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602635    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602637    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602640    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602642    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602644    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602647    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602650    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602652    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602655    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602659    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602662    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602664    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602666    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.612234 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602669    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602671    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602674    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602676    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602679    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602681    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602683    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602691    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602694    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602696    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602698    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602701    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602703    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602707    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602710    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602713    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602715    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602718    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602720    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.612717 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602723    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602725    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602727    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602730    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602732    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602735    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602737    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602739    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602742    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602746    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602748    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602751    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602753    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602755    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602758    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602760    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602763    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.613207 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602765    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.602792    2542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602837    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602841    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602850    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602854    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602857    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602860    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602863    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602865    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602868    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602870    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602873    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602875    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602878    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602880    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.613634 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602883    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602886    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602889    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602891    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602894    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602896    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602899    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602903    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602907    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602909    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602912    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602914    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602917    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602919    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602922    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602924    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602927    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602929    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602932    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602934    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.614004 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602937    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602945    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602948    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602950    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602953    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602955    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602958    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602960    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602963    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602965    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602968    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602971    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602975    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602978    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602980    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602983    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602985    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602987    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602990    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.614511 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602994    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.602998    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603001    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603004    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603006    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603009    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603011    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603014    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603016    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603019    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603021    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603024    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603026    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603029    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603031    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603039    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603041    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603044    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603047    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.614964 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603049    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603052    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603054    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603057    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603059    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603061    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603064    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603066    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603068    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603072    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603076    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603079    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603083    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603087    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603090    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603093    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603095    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603098    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603100    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.615563 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603103    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603106    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603108    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603111    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603113    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603116    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603118    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603121    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603123    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603126    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603135    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603138    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603141    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603144    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.603146    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.610541    2542 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.610558    2542 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.611409    2542 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.611417    2542 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611478    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.616009 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611484    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611488    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611491    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611494    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611497    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611500    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611503    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611506    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611510    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611514    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611517    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611520    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611522    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611525    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611528    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611530    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611533    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611535    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611538    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.616612 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611540    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611543    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611546    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611549    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611551    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611554    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611556    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611559    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611562    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611566    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611569    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611571    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611574    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611576    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611579    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611581    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611584    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611586    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611589    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611592    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.617079 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611595    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611597    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611600    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611602    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611605    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611607    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611610    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611612    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611615    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611617    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611620    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611623    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611625    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611628    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611630    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611633    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611635    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611637    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611640    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611643    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.617595 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611646    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611648    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611652    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611655    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611657    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611660    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611662    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611665    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611667    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611670    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611672    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611674    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611677    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611679    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611682    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611685    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611687    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611690    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611692    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611694    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.618068 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611697    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611700    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611702    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611705    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611707    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611710    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611712    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611715    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611717    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611719    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611722    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611725    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611727    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611730    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611732    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611739    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611743    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611747    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611750    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.618653 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611752    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611755    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611757    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611760    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611762    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611765    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611767    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.611795    2542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611939    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611943    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611947    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611950    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611952    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611955    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611957    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.619141 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611960    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611962    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611965    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611967    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611969    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611972    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611974    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611978    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611982    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611984    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611987    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611990    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611993    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611995    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.611998    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612001    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612003    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612006    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612008    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.619768 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612011    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612013    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612015    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612018    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612020    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612023    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612025    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612027    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612030    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612032    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612035    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612037    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612040    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612042    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612045    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612047    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612050    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612053    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612055    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612058    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.620421 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612060    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612063    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612065    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612068    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612070    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612072    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612075    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612077    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612080    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612082    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612085    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612087    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612089    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612092    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612094    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612097    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612099    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612101    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612104    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.620892 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612106    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612108    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612112    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612114    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612116    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612119    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612122    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612126    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612130    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612133    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612136    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612139    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612142    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612145    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612147    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612150    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612152    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612154    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612157    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.621624 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612159    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612161    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612164    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612167    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612169    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612172    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612174    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612176    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612179    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612181    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612184    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612186    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612189    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612191    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612193    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612196    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612198    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612201    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612203    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612205    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.622455 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612208    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612211    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.612233    2542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612268    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612273    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612277    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612279    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612282    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612285    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612287    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612290    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612292    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612295    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612298    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612300    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612302    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:56:34.622936 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612306    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612308    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612311    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612313    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612316    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612318    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612320    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612323    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612325    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612328    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612330    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612332    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612335    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612337    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612339    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612342    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612346    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612349    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612367    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:56:34.623305 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612369    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612372    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612375    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612377    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612380    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612382    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612385    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612387    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612390    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612392    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612395    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612397    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612399    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612402    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612405    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612408    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612410    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612413    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612415    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612418    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:56:34.623775 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612422    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612424    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612427    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612429    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612432    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612434    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612437    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612439    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612442    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612444    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612449    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612453    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612455    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612458    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612460    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612463    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612466    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612468    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612470    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612473    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:56:34.624247 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612475    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612478    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612480    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612482    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612485    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612487    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612490    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612492    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612495    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612498    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612500    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612502    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612505    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612507    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612511    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612514    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612517    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612520    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612523    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:56:34.624739 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612525    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612529    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612531    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612535    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612538    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612541    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612544    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612546    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612549    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612551    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612554    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612556    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612559    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612562    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:34.612564    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.612618    2542 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.615034    2542 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.616147    2542 server.go:1014] "Starting client certificate rotation"
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.616258    2542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:56:34.625487 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.616300    2542 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:56:34.625957 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.618486    2542 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:56:34.625957 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.621707    2542 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:56:34.635627 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.635604    2542 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:56:34.639762 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.639744    2542 log.go:25] "Validated CRI v1 image API"
Nov 02 13:56:34.640693 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.640660    2542 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:56:34.646031 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.646003    2542 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 94720379-0f7f-4a75-a6e8-b841fc761c57:/dev/nvme0n1p4 b6930021-2899-4cc4-83be-7668c49c215d:/dev/nvme0n1p3]
Nov 02 13:56:34.646153 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.646026    2542 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 13:56:34.651521 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.651387    2542 manager.go:217] Machine: {Timestamp:2026-11-02 13:56:34.64988138 +0000 UTC m=+0.370010613 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099792 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b74a1f06bea11e0e3162eea19f8a5 SystemUUID:ec2b74a1-f06b-ea11-e0e3-162eea19f8a5 BootID:49a54c6c-d663-49df-b46a-5975d2732a4a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:5e:60:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:5e:60:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:90:45:71:11:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:56:34.651521 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.651512    2542 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:56:34.651716 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.651628    2542 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:56:34.655288 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.655252    2542 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:56:34.655452 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.655290    2542 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:56:34.655504 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.655455    2542 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:56:34.655504 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.655465    2542 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:56:34.655562 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.655505    2542 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:56:34.655562 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.655524    2542 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:56:34.656807 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.656795    2542 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:56:34.661546 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.661532    2542 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:56:34.661867 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.661857    2542 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:56:34.662883 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.662872    2542 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:56:34.662914 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.662888    2542 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:56:34.662914 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.662900    2542 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:56:34.662914 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.662910    2542 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:56:34.663055 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.662918    2542 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:56:34.663339 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.663317    2542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:56:34.665977 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.665956    2542 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:56:34.673214 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.673199    2542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:56:34.674122 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674109    2542 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:56:34.674171 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674141    2542 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:56:34.674304 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674294    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:56:34.674340 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674310    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:56:34.674340 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674316    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:56:34.674340 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674321    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:56:34.674340 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674327    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:56:34.674340 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.674332    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:56:34.675009 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675000    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:56:34.675038 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675010    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:56:34.675038 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675017    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:56:34.675038 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675025    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:56:34.675038 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675033    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:56:34.675238 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675042    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:56:34.675926 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675917    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:56:34.675926 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.675927    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:56:34.680177 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.680163    2542 server.go:1263] "Started kubelet"
Nov 02 13:56:34.680287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.680259    2542 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:56:34.680341 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.680264    2542 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:56:34.680410 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.680345    2542 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:56:34.681147 ip-10-0-129-115 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:56:34.681856 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.681812    2542 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:56:34.682535 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.682503    2542 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:56:34.687602 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.687579    2542 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 13:56:34.689265 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.689249    2542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:56:34.689348 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.689249    2542 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:56:34.689591 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.689566    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:34.690134 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690111    2542 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:56:34.690134 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690113    2542 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:56:34.690134 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690131    2542 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:56:34.690490 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690472    2542 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:56:34.690490 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690490    2542 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:56:34.690664 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690650    2542 factory.go:55] Registering systemd factory
Nov 02 13:56:34.690747 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690666    2542 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:56:34.690926 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690909    2542 factory.go:153] Registering CRI-O factory
Nov 02 13:56:34.690971 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690929    2542 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:56:34.691015 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.690984    2542 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:56:34.691044 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.691018    2542 factory.go:103] Registering Raw factory
Nov 02 13:56:34.691044 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.691029    2542 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:56:34.692274 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.692251    2542 manager.go:319] Starting recovery of all containers
Nov 02 13:56:34.705259 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.705020    2542 manager.go:324] Recovery completed
Nov 02 13:56:34.706548 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.706523    2542 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 13:56:34.709680 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.709667    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.711707 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.711691    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.711764 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.711722    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.711764 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.711732    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.712239 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712225    2542 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:56:34.712239 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712232    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.712391 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712236    2542 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:56:34.712391 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712272    2542 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:56:34.712952 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712932    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.713024 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712970    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.713024 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.712980    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.714511 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.714485    2542 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 13:56:34.714598 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.714525    2542 policy_none.go:50] "Start"
Nov 02 13:56:34.714598 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.714540    2542 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:56:34.714598 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.714553    2542 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:56:34.714598 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.714591    2542 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:56:34.715547 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.714372    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-115.ec2.internal.18e43ef9e92d34b6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-115.ec2.internal,},FirstTimestamp:2026-11-02 13:56:34.68013279 +0000 UTC m=+0.400262023,LastTimestamp:2026-11-02 13:56:34.68013279 +0000 UTC m=+0.400262023,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-115.ec2.internal,}"
Nov 02 13:56:34.716613 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.716601    2542 policy_none.go:44] "Start"
Nov 02 13:56:34.732152 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.732081    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-115.ec2.internal.18e43ef9eb0eff38  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-115.ec2.internal,},FirstTimestamp:2026-11-02 13:56:34.711707448 +0000 UTC m=+0.431836683,LastTimestamp:2026-11-02 13:56:34.711707448 +0000 UTC m=+0.431836683,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-115.ec2.internal,}"
Nov 02 13:56:34.732651 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.732628    2542 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 13:56:34.732713 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.732629    2542 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:56:34.743472 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.743449    2542 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:56:34.744291 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.744270    2542 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:56:34.744389 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.744337    2542 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxjc8"
Nov 02 13:56:34.753437 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.753415    2542 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxjc8"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.754093    2542 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.758936    2542 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.758912    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-115.ec2.internal.18e43ef9eb0f4929  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-115.ec2.internal,},FirstTimestamp:2026-11-02 13:56:34.711726377 +0000 UTC m=+0.431855610,LastTimestamp:2026-11-02 13:56:34.711726377 +0000 UTC m=+0.431855610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-115.ec2.internal,}"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.759001    2542 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.759012    2542 server.go:87] "Starting device plugin registration server"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.759301    2542 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.759313    2542 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.759482    2542 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.759559    2542 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.759567    2542 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.760153    2542 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.760204    2542 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.772463    2542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.773988    2542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.774002    2542 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.774026    2542 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.774072    2542 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 13:56:34.776463 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.775275    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-115.ec2.internal.18e43ef9eb0f6edf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-115.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-115.ec2.internal,},FirstTimestamp:2026-11-02 13:56:34.711736031 +0000 UTC m=+0.431865264,LastTimestamp:2026-11-02 13:56:34.711736031 +0000 UTC m=+0.431865264,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-115.ec2.internal,}"
Nov 02 13:56:34.815668 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.815646    2542 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:56:34.859916 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.859849    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.860975 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.860959    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.861063 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.860988    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.861063 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.861003    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.861063 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.861031    2542 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.875150 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.875127    2542 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"]
Nov 02 13:56:34.875204 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.875197    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.876015 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.875997    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.876088 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.876027    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.876088 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.876040    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.878298 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.878286    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.878440 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.878425    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.878828 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.878808    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.879018 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.879003    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.879094 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.879033    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.879094 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.879055    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.879529 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.879515    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.879584 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.879540    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.879584 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.879549    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.881187 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.881174    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.881235 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.881199    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:56:34.883544 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.883510    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:56:34.883897 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.883880    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:56:34.883943 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.883918    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:56:34.886424 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.886407    2542 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.886519 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.886439    2542 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-115.ec2.internal\": node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:34.890860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.890842    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.890915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.890867    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aff7ff2bf177bc815214d878993baa38-config\") pod \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" (UID: \"aff7ff2bf177bc815214d878993baa38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.890915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.890884    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.912516 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.912486    2542 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-115.ec2.internal\" not found" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.919710 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.919691    2542 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-115.ec2.internal\" not found" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.938597 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:34.938578    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:34.991833 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.991807    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aff7ff2bf177bc815214d878993baa38-config\") pod \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" (UID: \"aff7ff2bf177bc815214d878993baa38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.991833 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.991834    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.992010 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.991851    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.992010 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.991904    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aff7ff2bf177bc815214d878993baa38-config\") pod \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" (UID: \"aff7ff2bf177bc815214d878993baa38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.992010 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.991917    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:34.992010 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:34.991905    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:35.038980 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.038950    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.140006 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.139934    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.217160 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.217133    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:35.224093 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.224073    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:35.240970 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.240944    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.341908 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.341868    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.442439 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.442408    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.542911 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.542866    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.615401 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.615365    2542 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 13:56:35.616093 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.615524    2542 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 13:56:35.643614 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:35.643582    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:56:35.703793 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.703723    2542 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:56:35.734748 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.734714    2542 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:56:35.760453 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.760412    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:51:35 +0000 UTC" deadline="2028-06-09 09:38:22.563136188 +0000 UTC"
Nov 02 13:56:35.760453 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.760441    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14035h41m46.802698139s"
Nov 02 13:56:35.790533 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.790490    2542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:35.812496 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:35.812467    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7ff2bf177bc815214d878993baa38.slice/crio-86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168 WatchSource:0}: Error finding container 86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168: Status 404 returned error can't find the container with id 86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168
Nov 02 13:56:35.813745 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.813722    2542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:56:35.814167 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:35.814145    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15570f366559ffb67f868c74c61ae83.slice/crio-1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb WatchSource:0}: Error finding container 1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb: Status 404 returned error can't find the container with id 1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb
Nov 02 13:56:35.815906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.815888    2542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:56:35.819250 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.819236    2542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:56:35.835952 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.835924    2542 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:56:35.837614 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:35.837599    2542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:56:36.663916 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.663888    2542 apiserver.go:52] "Watching apiserver"
Nov 02 13:56:36.689628 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.689592    2542 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 13:56:36.695562 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.695534    2542 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:56:36.695964 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.695934    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-t2wc7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal","openshift-multus/multus-88ssr","openshift-ovn-kubernetes/ovnkube-node-g557m","kube-system/global-pull-secret-syncer-csrlb","openshift-network-diagnostics/network-check-target-8cskr","openshift-image-registry/node-ca-wn8xj","kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg","openshift-multus/multus-additional-cni-plugins-wqm7n","openshift-multus/network-metrics-daemon-zjfbv","openshift-network-operator/iptables-alerter-m622v","kube-system/konnectivity-agent-2sbs7","openshift-cluster-node-tuning-operator/tuned-9xdrt"]
Nov 02 13:56:36.698623 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.698604    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.700930 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.700910    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.703230 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.703213    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.705896 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.705874    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.708371 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.708341    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.708486 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.708449    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:36.711250 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.711141    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:36.711250 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.711215    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:36.713678 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.713660    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.716728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.716333    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.719146 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.719125    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.723877 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.723854    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.727325 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.727306    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:36.727460 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.727402    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:36.729669 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.729650    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.742021 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.741474    2542 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:56:36.742021 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.741549    2542 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-trrtl"
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750575    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750579    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750638    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750637    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750641    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750666    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750708    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750708    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750779    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750805    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750854    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750889    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750907    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750916    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750931    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750936    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750958    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750965    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mgxc\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750931    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750973    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.750999    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlrs4\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.751012    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.751023 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.751040    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.758590 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.758556    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.761344 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.761309    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cd6m4\""
Nov 02 13:56:36.763838 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.763816    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75928\""
Nov 02 13:56:36.763942 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.763912    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:56:36.764010 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.763967    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2zrsm\""
Nov 02 13:56:36.764104 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764087    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:56:36.764168 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764146    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4wpm\""
Nov 02 13:56:36.764311 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764296    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5vzv\""
Nov 02 13:56:36.764489 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764475    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kftg9\""
Nov 02 13:56:36.764572 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764555    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:56:36.764634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764604    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gjwh\""
Nov 02 13:56:36.764715 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764702    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:56:36.764801 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764786    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:56:36.764914 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.764886    2542 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-trrtl"
Nov 02 13:56:36.781070 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.781010    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" event={"ID":"b15570f366559ffb67f868c74c61ae83","Type":"ContainerStarted","Data":"1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb"}
Nov 02 13:56:36.782756 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.782698    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal" event={"ID":"aff7ff2bf177bc815214d878993baa38","Type":"ContainerStarted","Data":"86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168"}
Nov 02 13:56:36.791680 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.791655    2542 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:56:36.803776 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803743    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-tuned\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803792    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803816    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9047a53-882d-4708-945a-663dcdc7dc9a-agent-certs\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803840    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9047a53-882d-4708-945a-663dcdc7dc9a-konnectivity-ca\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803861    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803878    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803895    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b19899c7-dc0f-439b-aa44-5948528cd810-ovn-node-metrics-cert\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.803923 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803912    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzfl\" (UniqueName: \"kubernetes.io/projected/7f928dc3-3c2e-4baa-b29b-de0a84eed156-kube-api-access-vfzfl\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803932    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-kubelet\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.803977    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-hostroot\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804003    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-multus-certs\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804020    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-script-lib\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804036    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj42k\" (UniqueName: \"kubernetes.io/projected/b19899c7-dc0f-439b-aa44-5948528cd810-kube-api-access-hj42k\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804057    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-modprobe-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804076    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-run\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804097    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804115    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-config\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804138    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-host\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804154    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hdn\" (UniqueName: \"kubernetes.io/projected/6e9884e1-a236-4925-9875-53f6b1e64ba0-kube-api-access-m9hdn\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.804184 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804169    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804190    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-system-cni-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804205    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-dbus\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804223    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-tmp-dir\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804243    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysconfig\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804259    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-var-lib-kubelet\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804309    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-netns\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804333    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-systemd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804348    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-ovn\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804367    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-netd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804389    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-systemd\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804415    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-tmp\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804460    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ccx5\" (UniqueName: \"kubernetes.io/projected/cec63cc5-8483-4e4f-aca4-2160dab7a670-kube-api-access-8ccx5\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804489    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.804592 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804534    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n479m\" (UniqueName: \"kubernetes.io/projected/6d5593ba-0e81-4d12-afbc-73019f33d7dd-kube-api-access-n479m\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804619    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-systemd-units\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804744    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-etc-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804791    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlxx\" (UniqueName: \"kubernetes.io/projected/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-kube-api-access-xdlxx\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804846    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd07b61c-b867-47ee-ae89-d324cbcaf032-serviceca\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804929    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-cnibin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804954    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.804973    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-device-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805022    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-multus\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805070    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-conf-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805108    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-daemon-config\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805141    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-node-log\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.805194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805186    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805224    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805299    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-lib-modules\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805376    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-iptables-alerter-script\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805412    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-netns\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805433    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-bin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805497    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805550    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-slash\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805595    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-socket-dir-parent\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.805791 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.805631    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cnibin\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.806157 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806132    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.806279 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806217    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-bin\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.806407 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806292    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.806407 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806314    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.806407 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806361    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-host-slash\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.806407 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806404    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd07b61c-b867-47ee-ae89-d324cbcaf032-host\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806424    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpdb\" (UniqueName: \"kubernetes.io/projected/dd07b61c-b867-47ee-ae89-d324cbcaf032-kube-api-access-5jpdb\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806452    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806517    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-sys\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806540    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-system-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806566    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5mh\" (UniqueName: \"kubernetes.io/projected/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kube-api-access-5n5mh\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806588    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806607    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-kubelet\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.806634 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806626    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-log-socket\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806643    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-env-overrides\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806662    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264tr\" (UniqueName: \"kubernetes.io/projected/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-kube-api-access-264tr\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806741    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-cni-binary-copy\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806769    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-k8s-cni-cncf-io\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806788    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-hosts-file\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806808    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-kubernetes\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806824    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-conf\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806841    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806858    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-etc-kubernetes\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806881    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806912    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-var-lib-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.807033 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.806937    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-kubelet-config\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.907407 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907307    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.907407 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907392    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n479m\" (UniqueName: \"kubernetes.io/projected/6d5593ba-0e81-4d12-afbc-73019f33d7dd-kube-api-access-n479m\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.907626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907433    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-systemd-units\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.907626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907461    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-etc-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.907626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907474    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.907626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907566    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-systemd-units\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.907626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907571    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlxx\" (UniqueName: \"kubernetes.io/projected/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-kube-api-access-xdlxx\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.907626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907621    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-etc-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907618    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd07b61c-b867-47ee-ae89-d324cbcaf032-serviceca\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907709    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-cnibin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907761    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907763    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-cnibin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907806    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-device-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907834    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-multus\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.907906 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907862    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-conf-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907911    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-device-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907927    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-multus\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907934    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907956    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-daemon-config\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907966    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-conf-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.907978    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-node-log\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908006    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908033    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908056    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-node-log\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908077    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-lib-modules\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908121    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-iptables-alerter-script\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908128    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908140    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-netns\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908167    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-bin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908196    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908223    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-slash\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.908253 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908219    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-netns\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908259    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-socket-dir-parent\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908294    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd07b61c-b867-47ee-ae89-d324cbcaf032-serviceca\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908309    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-bin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908357    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-socket-dir-parent\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908603    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cnibin\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908331    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cnibin\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.908860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908848    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.909091 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908884    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-daemon-config\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.909091 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908923    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-bin\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.909091 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908281    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-lib-modules\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.909091 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908988    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-bin\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.909229 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909153    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.909229 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909173    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-iptables-alerter-script\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.909229 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909201    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.909325 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.909272    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:36.909325 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.908371    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-slash\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.909400 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909375    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.909491 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909445    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.909572 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909516    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-host-slash\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.909572 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909543    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd07b61c-b867-47ee-ae89-d324cbcaf032-host\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.909655 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909574    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpdb\" (UniqueName: \"kubernetes.io/projected/dd07b61c-b867-47ee-ae89-d324cbcaf032-kube-api-access-5jpdb\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.909655 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909568    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.909655 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909609    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.909655 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909641    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-sys\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.909785 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909669    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd07b61c-b867-47ee-ae89-d324cbcaf032-host\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.909785 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909672    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-system-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.909785 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909728    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5mh\" (UniqueName: \"kubernetes.io/projected/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kube-api-access-5n5mh\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.909785 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909732    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-system-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.909900 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909788    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.909900 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909821    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-kubelet\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.909900 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909858    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-log-socket\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.909989 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909911    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-env-overrides\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.910022 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.909974    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264tr\" (UniqueName: \"kubernetes.io/projected/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-kube-api-access-264tr\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.910052 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910040    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-cni-binary-copy\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.910087 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910067    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-k8s-cni-cncf-io\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.910122 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910096    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-sys\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.910122 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910099    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-hosts-file\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.910185 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910146    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-hosts-file\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.910215 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910148    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-kubernetes\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910260    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-conf\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910268    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-kubelet\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.910310    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret podName:519bdf1e-bdde-4785-a71c-74c0d80ea527 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.410263056 +0000 UTC m=+3.130392300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret") pod "global-pull-secret-syncer-csrlb" (UID: "519bdf1e-bdde-4785-a71c-74c0d80ea527") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910347    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-kubernetes\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910452    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-conf\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910522    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-log-socket\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910597    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910626    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-etc-kubernetes\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910663    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910696    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-var-lib-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910734    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-var-lib-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910784    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910794    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910820    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-etc-kubernetes\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910872    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-host-slash\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910874    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.914359 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.910996    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-cni-binary-copy\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911070    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-k8s-cni-cncf-io\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911087    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-kubelet-config\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911189    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-tuned\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911209    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-kubelet-config\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911230    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911262    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9047a53-882d-4708-945a-663dcdc7dc9a-agent-certs\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911301    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9047a53-882d-4708-945a-663dcdc7dc9a-konnectivity-ca\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911288    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-env-overrides\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911342    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911405    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911444    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b19899c7-dc0f-439b-aa44-5948528cd810-ovn-node-metrics-cert\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911486    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzfl\" (UniqueName: \"kubernetes.io/projected/7f928dc3-3c2e-4baa-b29b-de0a84eed156-kube-api-access-vfzfl\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911525    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-kubelet\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911567    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-hostroot\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911608    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911616    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-multus-certs\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915154 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911688    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-multus-certs\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911697    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-script-lib\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911709    2542 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911759    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-hostroot\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911772    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-kubelet\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911759    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj42k\" (UniqueName: \"kubernetes.io/projected/b19899c7-dc0f-439b-aa44-5948528cd810-kube-api-access-hj42k\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911828    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-modprobe-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911865    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-run\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911909    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911950    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-config\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.911999    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-host\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912032    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hdn\" (UniqueName: \"kubernetes.io/projected/6e9884e1-a236-4925-9875-53f6b1e64ba0-kube-api-access-m9hdn\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912086    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912161    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912237    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912352    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-modprobe-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912461    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-run\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912513    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-system-cni-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.915976 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912576    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-dbus\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912627    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-tmp-dir\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912721    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysconfig\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912768    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-var-lib-kubelet\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912855    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-netns\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912891    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-systemd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.912948    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-ovn\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913012    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-netd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913042    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-systemd\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913094    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-tmp\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913147    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-script-lib\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913190    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccx5\" (UniqueName: \"kubernetes.io/projected/cec63cc5-8483-4e4f-aca4-2160dab7a670-kube-api-access-8ccx5\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913391    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-config\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913502    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-host\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913568    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9047a53-882d-4708-945a-663dcdc7dc9a-konnectivity-ca\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913656    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-systemd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913656    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-var-lib-kubelet\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913810    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysconfig\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.916922 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.913955    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914019    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-tmp-dir\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914028    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-netns\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.914057    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914115    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-system-cni-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914163    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-ovn\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914193    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-dbus\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.914265    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.414238627 +0000 UTC m=+3.134367870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914253    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-systemd\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914294    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-netd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.914649    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.917876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.915971    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-tuned\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.918782 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.918369    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-tmp\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:36.918782 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.918565    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b19899c7-dc0f-439b-aa44-5948528cd810-ovn-node-metrics-cert\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.920264 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.920119    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9047a53-882d-4708-945a-663dcdc7dc9a-agent-certs\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:36.931880 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.931228    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:56:36.931880 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.931255    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:56:36.931880 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.931272    2542 projected.go:196] Error preparing data for projected volume kube-api-access-t5ggh for pod openshift-network-diagnostics/network-check-target-8cskr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:36.931880 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:36.931331    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh podName:8e854bb3-3994-496f-bc04-3ace19a132a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.431312173 +0000 UTC m=+3.151441417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t5ggh" (UniqueName: "kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh") pod "network-check-target-8cskr" (UID: "8e854bb3-3994-496f-bc04-3ace19a132a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:36.934653 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.934598    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n479m\" (UniqueName: \"kubernetes.io/projected/6d5593ba-0e81-4d12-afbc-73019f33d7dd-kube-api-access-n479m\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:36.935228 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.935179    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpdb\" (UniqueName: \"kubernetes.io/projected/dd07b61c-b867-47ee-ae89-d324cbcaf032-kube-api-access-5jpdb\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:36.937513 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.937483    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264tr\" (UniqueName: \"kubernetes.io/projected/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-kube-api-access-264tr\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:36.937846 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.937826    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5mh\" (UniqueName: \"kubernetes.io/projected/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kube-api-access-5n5mh\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:36.939301 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.939257    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzfl\" (UniqueName: \"kubernetes.io/projected/7f928dc3-3c2e-4baa-b29b-de0a84eed156-kube-api-access-vfzfl\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:36.939873 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.939844    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlxx\" (UniqueName: \"kubernetes.io/projected/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-kube-api-access-xdlxx\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:36.940006 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.939971    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj42k\" (UniqueName: \"kubernetes.io/projected/b19899c7-dc0f-439b-aa44-5948528cd810-kube-api-access-hj42k\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:36.942095 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.942068    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hdn\" (UniqueName: \"kubernetes.io/projected/6e9884e1-a236-4925-9875-53f6b1e64ba0-kube-api-access-m9hdn\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:56:36.943795 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:36.943770    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccx5\" (UniqueName: \"kubernetes.io/projected/cec63cc5-8483-4e4f-aca4-2160dab7a670-kube-api-access-8ccx5\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:37.014639 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.014606    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:56:37.021870 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.021812    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:37.025558 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.025104    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd07b61c_b867_47ee_ae89_d324cbcaf032.slice/crio-aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed WatchSource:0}: Error finding container aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed: Status 404 returned error can't find the container with id aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed
Nov 02 13:56:37.031374 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.031344    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9047a53_882d_4708_945a_663dcdc7dc9a.slice/crio-14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564 WatchSource:0}: Error finding container 14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564: Status 404 returned error can't find the container with id 14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564
Nov 02 13:56:37.033684 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.033662    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-88ssr"
Nov 02 13:56:37.046430 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.046398    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9884e1_a236_4925_9875_53f6b1e64ba0.slice/crio-ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d WatchSource:0}: Error finding container ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d: Status 404 returned error can't find the container with id ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d
Nov 02 13:56:37.063234 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.063030    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:56:37.066509 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.066482    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:56:37.068233 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.068206    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:37.070580 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.070552    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:56:37.076656 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.076369    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb4aab9_6e60_4c30_bdd5_eb34ae6ceee3.slice/crio-82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d WatchSource:0}: Error finding container 82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d: Status 404 returned error can't find the container with id 82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d
Nov 02 13:56:37.083283 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.083255    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25f0e9a_bcc1_4ce1_9361_b98df140e5f3.slice/crio-67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648 WatchSource:0}: Error finding container 67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648: Status 404 returned error can't find the container with id 67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648
Nov 02 13:56:37.084946 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.084923    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19899c7_dc0f_439b_aa44_5948528cd810.slice/crio-b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97 WatchSource:0}: Error finding container b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97: Status 404 returned error can't find the container with id b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97
Nov 02 13:56:37.088834 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.088810    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:56:37.091268 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.091212    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:56:37.092126 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.092100    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec63cc5_8483_4e4f_aca4_2160dab7a670.slice/crio-edef2fda5d2e352c3112b4c142285f5d8880469f03228316029a3f0b141f61f8 WatchSource:0}: Error finding container edef2fda5d2e352c3112b4c142285f5d8880469f03228316029a3f0b141f61f8: Status 404 returned error can't find the container with id edef2fda5d2e352c3112b4c142285f5d8880469f03228316029a3f0b141f61f8
Nov 02 13:56:37.101840 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.101812    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5593ba_0e81_4d12_afbc_73019f33d7dd.slice/crio-06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830 WatchSource:0}: Error finding container 06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830: Status 404 returned error can't find the container with id 06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830
Nov 02 13:56:37.105907 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:56:37.105807    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ecb7ef_ae15_4411_b6f2_5c515d9e3284.slice/crio-4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7 WatchSource:0}: Error finding container 4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7: Status 404 returned error can't find the container with id 4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7
Nov 02 13:56:37.178557 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.178532    2542 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:56:37.417045 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.416876    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:37.417045 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.416944    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:37.417045 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.417003    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:37.417378 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.417083    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret podName:519bdf1e-bdde-4785-a71c-74c0d80ea527 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.417063188 +0000 UTC m=+4.137192428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret") pod "global-pull-secret-syncer-csrlb" (UID: "519bdf1e-bdde-4785-a71c-74c0d80ea527") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:37.417378 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.417092    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:37.417378 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.417151    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.417131676 +0000 UTC m=+4.137260914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:37.518742 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.517434    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:37.518742 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.517619    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:56:37.518742 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.517643    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:56:37.518742 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.517656    2542 projected.go:196] Error preparing data for projected volume kube-api-access-t5ggh for pod openshift-network-diagnostics/network-check-target-8cskr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:37.518742 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:37.517724    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh podName:8e854bb3-3994-496f-bc04-3ace19a132a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.517695455 +0000 UTC m=+4.237824694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5ggh" (UniqueName: "kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh") pod "network-check-target-8cskr" (UID: "8e854bb3-3994-496f-bc04-3ace19a132a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:37.769642 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.769550    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-08-19 08:15:08.16141645 +0000 UTC"
Nov 02 13:56:37.769642 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.769595    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15738h18m30.391827035s"
Nov 02 13:56:37.814558 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.814404    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg" event={"ID":"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3","Type":"ContainerStarted","Data":"67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648"}
Nov 02 13:56:37.826876 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.826789    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t2wc7" event={"ID":"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3","Type":"ContainerStarted","Data":"82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d"}
Nov 02 13:56:37.830952 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.830909    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-88ssr" event={"ID":"6e9884e1-a236-4925-9875-53f6b1e64ba0","Type":"ContainerStarted","Data":"ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d"}
Nov 02 13:56:37.834781 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.834727    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wn8xj" event={"ID":"dd07b61c-b867-47ee-ae89-d324cbcaf032","Type":"ContainerStarted","Data":"aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed"}
Nov 02 13:56:37.839650 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.839581    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal" event={"ID":"aff7ff2bf177bc815214d878993baa38","Type":"ContainerStarted","Data":"05718734b1cb9787b07ab7b872da7ea7e54bdcfa4b390054b422a607b9853b72"}
Nov 02 13:56:37.851894 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.851863    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerStarted","Data":"06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830"}
Nov 02 13:56:37.876871 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.876749    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2sbs7" event={"ID":"f9047a53-882d-4708-945a-663dcdc7dc9a","Type":"ContainerStarted","Data":"14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564"}
Nov 02 13:56:37.905894 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.905851    2542 generic.go:356] "Generic (PLEG): container finished" podID="b15570f366559ffb67f868c74c61ae83" containerID="f2eb5f4483a84ca34d675fc4a62d37c4dcab5b7eb9130f27d8df84a325f7dfa8" exitCode=0
Nov 02 13:56:37.906080 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.905953    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" event={"ID":"b15570f366559ffb67f868c74c61ae83","Type":"ContainerDied","Data":"f2eb5f4483a84ca34d675fc4a62d37c4dcab5b7eb9130f27d8df84a325f7dfa8"}
Nov 02 13:56:37.920347 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.920310    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m622v" event={"ID":"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284","Type":"ContainerStarted","Data":"4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7"}
Nov 02 13:56:37.947226 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.947164    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal" podStartSLOduration=1.947142113 podStartE2EDuration="1.947142113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:37.87974828 +0000 UTC m=+3.599877564" watchObservedRunningTime="2026-11-02 13:56:37.947142113 +0000 UTC m=+3.667271359"
Nov 02 13:56:37.949299 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.949266    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9xdrt" event={"ID":"cec63cc5-8483-4e4f-aca4-2160dab7a670","Type":"ContainerStarted","Data":"edef2fda5d2e352c3112b4c142285f5d8880469f03228316029a3f0b141f61f8"}
Nov 02 13:56:37.959563 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:37.959445    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97"}
Nov 02 13:56:38.428482 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.426939    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:38.428482 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.427039    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:38.428482 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.427165    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:38.428482 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.427225    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret podName:519bdf1e-bdde-4785-a71c-74c0d80ea527 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.427207355 +0000 UTC m=+6.147336597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret") pod "global-pull-secret-syncer-csrlb" (UID: "519bdf1e-bdde-4785-a71c-74c0d80ea527") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:38.428482 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.427669    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:38.428482 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.427726    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.427708234 +0000 UTC m=+6.147837477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:38.528637 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.527836    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:38.528637 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.528000    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:56:38.528637 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.528025    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:56:38.528637 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.528040    2542 projected.go:196] Error preparing data for projected volume kube-api-access-t5ggh for pod openshift-network-diagnostics/network-check-target-8cskr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:38.528637 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.528104    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh podName:8e854bb3-3994-496f-bc04-3ace19a132a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.528083697 +0000 UTC m=+6.248212937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5ggh" (UniqueName: "kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh") pod "network-check-target-8cskr" (UID: "8e854bb3-3994-496f-bc04-3ace19a132a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:38.772255 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.770613    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-03-28 12:06:12.030911086 +0000 UTC"
Nov 02 13:56:38.772255 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.770680    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12286h9m33.260236134s"
Nov 02 13:56:38.778345 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.777392    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:38.778345 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.777555    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:38.778345 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.777975    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:38.778345 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.778085    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:38.778345 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.778145    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:38.778345 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:38.778217    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:38.982283 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:38.982243    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" event={"ID":"b15570f366559ffb67f868c74c61ae83","Type":"ContainerStarted","Data":"cb23286c13f306a08be571f1b497b87fb5db18ea3f9a53ba71cc80c5315c3dda"}
Nov 02 13:56:40.006766 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.005518    2542 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="1f872a7480fb8c999e4e3c5caeb1fcc18b58049e15f7b95927b179b98856044d" exitCode=0
Nov 02 13:56:40.006766 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.006489    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerDied","Data":"1f872a7480fb8c999e4e3c5caeb1fcc18b58049e15f7b95927b179b98856044d"}
Nov 02 13:56:40.078123 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.078005    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" podStartSLOduration=4.077983089 podStartE2EDuration="4.077983089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:39.023593649 +0000 UTC m=+4.743722897" watchObservedRunningTime="2026-11-02 13:56:40.077983089 +0000 UTC m=+5.798112335"
Nov 02 13:56:40.444792 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.444752    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:40.444991 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.444827    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:40.444991 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.444949    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:40.445119 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.445017    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.444998247 +0000 UTC m=+10.165127486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:40.445119 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.445084    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:40.445119 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.445116    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret podName:519bdf1e-bdde-4785-a71c-74c0d80ea527 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.445105738 +0000 UTC m=+10.165234975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret") pod "global-pull-secret-syncer-csrlb" (UID: "519bdf1e-bdde-4785-a71c-74c0d80ea527") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:40.546067 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.546029    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:40.546339 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.546258    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:56:40.546339 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.546283    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:56:40.546339 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.546296    2542 projected.go:196] Error preparing data for projected volume kube-api-access-t5ggh for pod openshift-network-diagnostics/network-check-target-8cskr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:40.546566 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.546348    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh podName:8e854bb3-3994-496f-bc04-3ace19a132a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.546333993 +0000 UTC m=+10.266463229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5ggh" (UniqueName: "kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh") pod "network-check-target-8cskr" (UID: "8e854bb3-3994-496f-bc04-3ace19a132a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:40.777899 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.777816    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:40.778061 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.777943    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:40.778402 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.778349    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:40.778543 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.778468    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:40.778543 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:40.778536    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:40.778667 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:40.778623    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:42.020027 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:42.019986    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2sbs7" event={"ID":"f9047a53-882d-4708-945a-663dcdc7dc9a","Type":"ContainerStarted","Data":"fc9c0c9ca6e1797baa153fdf0cb9e796acb71d134fa82f7a8be1b40244a7dff3"}
Nov 02 13:56:42.777506 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:42.776593    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:42.777506 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:42.776969    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:42.777506 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:42.777357    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:42.777506 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:42.777469    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:42.777882 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:42.777550    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:42.777882 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:42.777674    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:43.026704 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:43.026470    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wn8xj" event={"ID":"dd07b61c-b867-47ee-ae89-d324cbcaf032","Type":"ContainerStarted","Data":"a7e259a2bfffd59e59eb097b72b756bb46196211585a74b6a1765206cc8f43ec"}
Nov 02 13:56:43.070739 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:43.069536    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2sbs7" podStartSLOduration=4.15189998 podStartE2EDuration="8.069514658s" totalImagesPullingTime="3.917614678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:42.064800688 +0000 UTC m=+7.784929934" watchObservedRunningTime="2026-11-02 13:56:43.069514658 +0000 UTC m=+8.789643905"
Nov 02 13:56:44.032565 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.032523    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg" event={"ID":"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3","Type":"ContainerStarted","Data":"eaaadf53725f3c2d63540092f8d464a7c0109a4d153f79bbf5ddaf00a418b507"}
Nov 02 13:56:44.482141 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.482089    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:44.482324 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.482169    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:44.482324 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.482315    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:44.482474 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.482415    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.482366289 +0000 UTC m=+18.202495528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:44.482923 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.482892    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:44.483030 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.482951    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret podName:519bdf1e-bdde-4785-a71c-74c0d80ea527 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.482934743 +0000 UTC m=+18.203063994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret") pod "global-pull-secret-syncer-csrlb" (UID: "519bdf1e-bdde-4785-a71c-74c0d80ea527") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:44.582809 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.582771    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:44.583094 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.583073    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:56:44.583172 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.583099    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:56:44.583172 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.583113    2542 projected.go:196] Error preparing data for projected volume kube-api-access-t5ggh for pod openshift-network-diagnostics/network-check-target-8cskr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:44.583244 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.583172    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh podName:8e854bb3-3994-496f-bc04-3ace19a132a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.583155596 +0000 UTC m=+18.303284845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5ggh" (UniqueName: "kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh") pod "network-check-target-8cskr" (UID: "8e854bb3-3994-496f-bc04-3ace19a132a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:44.670119 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.670084    2542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:44.672101 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.672067    2542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:44.717223 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.717167    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wn8xj" podStartSLOduration=4.460191581 podStartE2EDuration="9.717145138s" totalImagesPullingTime="5.256953557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:43.070668366 +0000 UTC m=+8.790797603" watchObservedRunningTime="2026-11-02 13:56:44.717145138 +0000 UTC m=+10.437274383"
Nov 02 13:56:44.776231 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.775295    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:44.776231 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.775425    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:44.776231 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.775869    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:44.776231 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.775971    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:44.776231 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.776067    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:44.776231 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:44.776144    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:44.914136 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:44.914104    2542 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:56:45.042213 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:45.042128    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg" event={"ID":"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3","Type":"ContainerStarted","Data":"59dcd394c8e6b4d626f1af4262fba623b2ccb9b6a8e93c55c5f0895077b41782"}
Nov 02 13:56:45.766315 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:45.765806    2542 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:56:44.914129026Z","UUID":"5c02882d-b8aa-44fd-a3ff-289e6aa0f1dd","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:56:45.767965 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:45.767941    2542 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:56:45.768054 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:45.767977    2542 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:56:46.047111 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.047067    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t2wc7" event={"ID":"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3","Type":"ContainerStarted","Data":"899f96396604388cea7046a1d66b4605b44030dfec372815f7e1f3a029d414ee"}
Nov 02 13:56:46.049777 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.049743    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-88ssr" event={"ID":"6e9884e1-a236-4925-9875-53f6b1e64ba0","Type":"ContainerStarted","Data":"8c04c7a5a1ccedb80e9f8ec569180717b02353a95581ef574af871c793e029b3"}
Nov 02 13:56:46.053111 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.053080    2542 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="fc3f5e02fe25e504c5b4e8ceb1a2758146c8e413badc47b9830ad6cda928d6e8" exitCode=0
Nov 02 13:56:46.053228 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.053123    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerDied","Data":"fc3f5e02fe25e504c5b4e8ceb1a2758146c8e413badc47b9830ad6cda928d6e8"}
Nov 02 13:56:46.092639 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.092582    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t2wc7" podStartSLOduration=2.628562843 podStartE2EDuration="11.092564161s" totalImagesPullingTime="8.464001318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:46.091221396 +0000 UTC m=+11.811350646" watchObservedRunningTime="2026-11-02 13:56:46.092564161 +0000 UTC m=+11.812693407"
Nov 02 13:56:46.187307 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.186994    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-88ssr" podStartSLOduration=2.868044844 podStartE2EDuration="11.186976013s" totalImagesPullingTime="8.318931169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:46.186522907 +0000 UTC m=+11.906652153" watchObservedRunningTime="2026-11-02 13:56:46.186976013 +0000 UTC m=+11.907105261"
Nov 02 13:56:46.777514 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.777392    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:46.777643 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:46.777592    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:46.777980 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.777956    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:46.778095 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:46.778065    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:46.778197 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:46.778143    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:46.778251 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:46.778226    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:47.059325 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:47.059218    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg" event={"ID":"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3","Type":"ContainerStarted","Data":"a8f046dae96f2380b982c1d147962915ac8ae54be383cc48970eb18f11ef1730"}
Nov 02 13:56:47.062775 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:47.062737    2542 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="5f9306f60a54349174445c9344744b0edd285036a4d9b4c13d018468cdba6116" exitCode=0
Nov 02 13:56:47.062928 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:47.062874    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerDied","Data":"5f9306f60a54349174445c9344744b0edd285036a4d9b4c13d018468cdba6116"}
Nov 02 13:56:47.066309 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:47.066285    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m622v" event={"ID":"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284","Type":"ContainerStarted","Data":"5954854d997e08a624309b350914f3c88cd530d03c133a0a55bcdceec89d671f"}
Nov 02 13:56:47.106057 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:47.106008    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg" podStartSLOduration=3.668466844 podStartE2EDuration="12.105993359s" totalImagesPullingTime="8.437526515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:47.105494794 +0000 UTC m=+12.825624040" watchObservedRunningTime="2026-11-02 13:56:47.105993359 +0000 UTC m=+12.826122604"
Nov 02 13:56:47.202353 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:47.202295    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m622v" podStartSLOduration=4.251709522 podStartE2EDuration="12.202278084s" totalImagesPullingTime="7.950568562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:47.149637906 +0000 UTC m=+12.869767152" watchObservedRunningTime="2026-11-02 13:56:47.202278084 +0000 UTC m=+12.922407331"
Nov 02 13:56:48.070515 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:48.070467    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerStarted","Data":"325eeeda5e88835d02e1c340afbe346a59b248b9791442367d72de67c84fc2b1"}
Nov 02 13:56:48.072199 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:48.072164    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9xdrt" event={"ID":"cec63cc5-8483-4e4f-aca4-2160dab7a670","Type":"ContainerStarted","Data":"742fcbcba5bd7665e0659eda5842d6197577f5c8272ac4ab450dc4458fc64128"}
Nov 02 13:56:48.775201 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:48.775167    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:48.775405 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:48.775211    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:48.775405 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:48.775185    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:48.775405 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:48.775303    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:48.775405 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:48.775390    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:48.775559 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:48.775452    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:49.076658 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:49.076575    2542 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="325eeeda5e88835d02e1c340afbe346a59b248b9791442367d72de67c84fc2b1" exitCode=0
Nov 02 13:56:49.077035 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:49.076668    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerDied","Data":"325eeeda5e88835d02e1c340afbe346a59b248b9791442367d72de67c84fc2b1"}
Nov 02 13:56:49.150726 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:49.150651    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9xdrt" podStartSLOduration=3.585763133 podStartE2EDuration="14.150632657s" totalImagesPullingTime="10.564869524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:48.163411089 +0000 UTC m=+13.883540330" watchObservedRunningTime="2026-11-02 13:56:49.150632657 +0000 UTC m=+14.870761899"
Nov 02 13:56:50.775299 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:50.775270    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:50.775299 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:50.775279    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:50.775711 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:50.775312    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:50.775711 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:50.775411    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:50.775711 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:50.775519    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:50.775711 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:50.775623    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:51.085081 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:51.085045    2542 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="72479cb6ee28775ff2c61dee512ee2b5a40f5183ed0a6c9a8329d0418e8b47de" exitCode=0
Nov 02 13:56:51.085254 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:51.085091    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerDied","Data":"72479cb6ee28775ff2c61dee512ee2b5a40f5183ed0a6c9a8329d0418e8b47de"}
Nov 02 13:56:52.090059 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.090023    2542 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="e80eeca17b5f8936d2d558ea077b01293cc85b611cd1fe917e047923d90a5b21" exitCode=0
Nov 02 13:56:52.090498 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.090101    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerDied","Data":"e80eeca17b5f8936d2d558ea077b01293cc85b611cd1fe917e047923d90a5b21"}
Nov 02 13:56:52.546701 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.546543    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:52.546812 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.546733    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:52.546812 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.546677    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:52.546908 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.546817    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret podName:519bdf1e-bdde-4785-a71c-74c0d80ea527 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.546798109 +0000 UTC m=+34.266927343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret") pod "global-pull-secret-syncer-csrlb" (UID: "519bdf1e-bdde-4785-a71c-74c0d80ea527") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:56:52.546908 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.546854    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:52.546908 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.546901    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.54688612 +0000 UTC m=+34.267015355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:56:52.647266 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.647200    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:52.647405 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.647343    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:56:52.647405 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.647375    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:56:52.647405 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.647385    2542 projected.go:196] Error preparing data for projected volume kube-api-access-t5ggh for pod openshift-network-diagnostics/network-check-target-8cskr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:52.647512 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.647433    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh podName:8e854bb3-3994-496f-bc04-3ace19a132a9 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.647418539 +0000 UTC m=+34.367547774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t5ggh" (UniqueName: "kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh") pod "network-check-target-8cskr" (UID: "8e854bb3-3994-496f-bc04-3ace19a132a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:56:52.774621 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.774589    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:52.774770 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.774627    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:52.774770 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:52.774601    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:52.774770 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.774696    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:52.774899 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.774848    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:52.774986 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:52.774966    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:53.095156 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.095114    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" event={"ID":"6d5593ba-0e81-4d12-afbc-73019f33d7dd","Type":"ContainerStarted","Data":"2a2f91d3697bc51a46f0b285f23300f182e9e6d40809ac361dd021801ec805ad"}
Nov 02 13:56:53.098021 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.097998    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"376f95d620448cc56783ef01f1335e192bfdb75d73772e7acd4116e1aac0e2cf"}
Nov 02 13:56:53.098118 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.098065    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"747f9683c0c680b8c22a0eab5aa9d3f23c631ed6f15ff335bb1cceb0468c8041"}
Nov 02 13:56:53.098118 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.098095    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"bef4187b90c77c2fea9678aa41c408c219fa2f623cf407244b2674e6a7b2f828"}
Nov 02 13:56:53.098192 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.098126    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"ae7d1660739957fea9665591eb71153825db5619fc22f29bab6ee5d20106f282"}
Nov 02 13:56:53.098192 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.098157    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"6c7ed8ca0474dd5c4598a3230a992db6d386d547fd69e5fcf57834b6622d9564"}
Nov 02 13:56:53.098192 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.098183    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"7f70046cd3e3d55d005ad977adfcdee73c8facc72e2e22b6376b4e919d121cc3"}
Nov 02 13:56:53.141956 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:53.141911    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wqm7n" podStartSLOduration=7.554878242 podStartE2EDuration="18.141899116s" totalImagesPullingTime="10.587020874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:53.141746228 +0000 UTC m=+18.861875497" watchObservedRunningTime="2026-11-02 13:56:53.141899116 +0000 UTC m=+18.862028357"
Nov 02 13:56:54.775415 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:54.775378    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:54.775853 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:54.775423    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:54.775853 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:54.775490    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:54.775853 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:54.775551    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:54.775853 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:54.775575    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:54.775853 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:54.775664    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:55.105248 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:55.105214    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"971a7043ca34b190c6628c525017cba7e62075a54102d4f9bf2ab09234b791c2"}
Nov 02 13:56:56.754425 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:56.754265    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:56.754898 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:56.754532    2542 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:56:56.754996 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:56.754979    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:56:56.774885 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:56.774856    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:56.775006 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:56.774856    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:56.775006 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:56.774949    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:56.775109 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:56.775048    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:56.775109 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:56.774863    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:56.775181 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:56.775148    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:57.113393 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:57.113266    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" event={"ID":"b19899c7-dc0f-439b-aa44-5948528cd810","Type":"ContainerStarted","Data":"323fae2ac2ea94e4d2f7d758f59c1ca76e02045889e3ab5dfed5f4c5b97ec38a"}
Nov 02 13:56:57.113670 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:57.113650    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:57.129643 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:57.129618    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:57.203520 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:57.203474    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" podStartSLOduration=7.023071716 podStartE2EDuration="22.20346122s" totalImagesPullingTime="15.180389504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:57.19128326 +0000 UTC m=+22.911412502" watchObservedRunningTime="2026-11-02 13:56:57.20346122 +0000 UTC m=+22.923590462"
Nov 02 13:56:58.117247 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.117219    2542 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:56:58.117764 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.117717    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:58.133754 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.133730    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:56:58.774497 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.774466    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:58.774659 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.774511    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:58.774659 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.774467    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:58.774659 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:58.774598    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:58.774779 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:58.774702    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:58.774818 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:58.774790    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:58.829711 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.829641    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-csrlb"]
Nov 02 13:56:58.831958 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.831772    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zjfbv"]
Nov 02 13:56:58.832569 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:58.832545    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8cskr"]
Nov 02 13:56:59.119723 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:59.119638    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:56:59.119723 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:59.119682    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:56:59.120250 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:59.119744    2542 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:56:59.120250 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:59.119772    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:56:59.120250 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:59.119772    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:56:59.120250 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:59.119894    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:56:59.120250 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:56:59.120246    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:56:59.465780 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:56:59.465734    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:00.774914 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:00.774882    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:00.775416 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:00.774929    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:00.775416 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:00.774887    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:00.775416 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:00.775030    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:57:00.775416 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:00.775141    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:57:00.775416 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:00.775204    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:57:01.134985 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:01.134903    2542 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g557m" podUID="b19899c7-dc0f-439b-aa44-5948528cd810" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 13:57:02.774662 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:02.774624    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:02.774662 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:02.774638    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:02.775105 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:02.774686    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:02.775105 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:02.774814    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:57:02.775105 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:02.774920    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:57:02.775105 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:02.774979    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:57:04.778626 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:04.778412    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:04.779058 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:04.778779    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:04.779058 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:04.778469    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:04.779058 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:04.778880    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8cskr" podUID="8e854bb3-3994-496f-bc04-3ace19a132a9"
Nov 02 13:57:04.779058 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:04.778959    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjfbv" podUID="7f928dc3-3c2e-4baa-b29b-de0a84eed156"
Nov 02 13:57:04.779205 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:04.779061    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csrlb" podUID="519bdf1e-bdde-4785-a71c-74c0d80ea527"
Nov 02 13:57:04.965031 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:04.964994    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:05.118382 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.118282    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeReady"
Nov 02 13:57:05.118545 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.118461    2542 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 13:57:05.141490 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.141460    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:05.179326 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.179294    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8r8vr"]
Nov 02 13:57:05.185306 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.185284    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bpnnm"]
Nov 02 13:57:05.185443 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.185418    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.189986 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.189970    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.201161 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.201135    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:57:05.201161 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.201156    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:57:05.201161 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.201147    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:57:05.201397 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.201263    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9cgsx\""
Nov 02 13:57:05.202076 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.202056    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8r8vr"]
Nov 02 13:57:05.207076 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.207056    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bpnnm"]
Nov 02 13:57:05.302125 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.302090    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:57:05.302287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.302174    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:57:05.302287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.302269    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9rw8\""
Nov 02 13:57:05.337105 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.337082    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de9d0ea2-cd91-46df-b5fb-f322312c3390-cert\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.337241 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.337113    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-config-volume\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.337241 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.337131    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.337241 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.337199    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brhbn\" (UniqueName: \"kubernetes.io/projected/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-kube-api-access-brhbn\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.337408 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.337272    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jfpn\" (UniqueName: \"kubernetes.io/projected/de9d0ea2-cd91-46df-b5fb-f322312c3390-kube-api-access-5jfpn\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.337408 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.337315    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-tmp-dir\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.438425 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438396    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de9d0ea2-cd91-46df-b5fb-f322312c3390-cert\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.438425 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438429    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-config-volume\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.438668 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438443    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.438668 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:05.438534    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:05.438668 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:05.438588    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.938569729 +0000 UTC m=+31.658698949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:05.438668 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438583    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brhbn\" (UniqueName: \"kubernetes.io/projected/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-kube-api-access-brhbn\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.438668 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438611    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jfpn\" (UniqueName: \"kubernetes.io/projected/de9d0ea2-cd91-46df-b5fb-f322312c3390-kube-api-access-5jfpn\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.438668 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438626    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-tmp-dir\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.438962 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.438925    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-tmp-dir\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.439209 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.439186    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-config-volume\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.442728 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.442700    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de9d0ea2-cd91-46df-b5fb-f322312c3390-cert\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.462161 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.462134    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brhbn\" (UniqueName: \"kubernetes.io/projected/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-kube-api-access-brhbn\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.463172 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.463153    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jfpn\" (UniqueName: \"kubernetes.io/projected/de9d0ea2-cd91-46df-b5fb-f322312c3390-kube-api-access-5jfpn\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.498124 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.498101    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:05.805059 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.804981    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8r8vr"]
Nov 02 13:57:05.811200 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:57:05.811173    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9d0ea2_cd91_46df_b5fb_f322312c3390.slice/crio-ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9 WatchSource:0}: Error finding container ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9: Status 404 returned error can't find the container with id ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9
Nov 02 13:57:05.943162 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:05.943129    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:05.943307 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:05.943229    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:05.943307 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:05.943285    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:06.943272239 +0000 UTC m=+32.663401462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:06.135705 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.135623    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8r8vr" event={"ID":"de9d0ea2-cd91-46df-b5fb-f322312c3390","Type":"ContainerStarted","Data":"ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9"}
Nov 02 13:57:06.774967 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.774932    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:06.774967 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.774964    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:06.775208 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.775183    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:06.790299 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.790276    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:57:06.790464 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.790440    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:57:06.802124 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.802102    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8ms52\""
Nov 02 13:57:06.802124 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.802117    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:57:06.802313 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.802251    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:57:06.802313 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.802255    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm2nq\""
Nov 02 13:57:06.802313 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.802273    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sczcr\""
Nov 02 13:57:06.949568 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:06.949528    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:06.950022 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:06.949680    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:06.950022 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:06.949759    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.94973812 +0000 UTC m=+34.669867360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:07.715217 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:07.715183    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:07.825722 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:07.825676    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:08.561153 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.561119    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:08.561629 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.561183    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:08.561629 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:08.561268    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:57:08.561629 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:08.561333    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:40.561314992 +0000 UTC m=+66.281444235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : secret "metrics-daemon-secret" not found
Nov 02 13:57:08.563625 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.563602    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:08.633319 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.633274    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:08.662051 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.662020    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:08.664875 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.664849    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:08.910594 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.910562    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:08.937495 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.937464    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-csrlb"]
Nov 02 13:57:08.942206 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:57:08.942180    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519bdf1e_bdde_4785_a71c_74c0d80ea527.slice/crio-6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4 WatchSource:0}: Error finding container 6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4: Status 404 returned error can't find the container with id 6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4
Nov 02 13:57:08.963945 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:08.963916    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:08.964043 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:08.964026    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:08.964102 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:08.964092    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:12.96407254 +0000 UTC m=+38.684201763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:09.144053 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:09.144015    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8r8vr" event={"ID":"de9d0ea2-cd91-46df-b5fb-f322312c3390","Type":"ContainerStarted","Data":"b8eb97d61debb0d5ff261b082304dd7e84f298395c27f2d424a4ab7f485ce9a3"}
Nov 02 13:57:09.145158 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:09.145132    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-csrlb" event={"ID":"519bdf1e-bdde-4785-a71c-74c0d80ea527","Type":"ContainerStarted","Data":"6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4"}
Nov 02 13:57:09.185550 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:09.185503    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8r8vr" podStartSLOduration=1.567145908 podStartE2EDuration="4.185490403s" totalImagesPullingTime="2.618344495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:09.184777363 +0000 UTC m=+34.904906607" watchObservedRunningTime="2026-11-02 13:57:09.185490403 +0000 UTC m=+34.905619644"
Nov 02 13:57:09.239772 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:09.239697    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8cskr"]
Nov 02 13:57:09.245663 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:57:09.245636    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e854bb3_3994_496f_bc04_3ace19a132a9.slice/crio-2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808 WatchSource:0}: Error finding container 2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808: Status 404 returned error can't find the container with id 2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808
Nov 02 13:57:10.149048 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.149008    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8cskr" event={"ID":"8e854bb3-3994-496f-bc04-3ace19a132a9","Type":"ContainerStarted","Data":"2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808"}
Nov 02 13:57:10.234558 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.234512    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:10.307015 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.306979    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6mkt2"]
Nov 02 13:57:10.324233 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.324199    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.342040 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.342014    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mkt2"]
Nov 02 13:57:10.345709 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.345684    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:57:10.352117 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.352096    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjjmq\""
Nov 02 13:57:10.371237 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.371204    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:57:10.371237 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.371221    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:57:10.371237 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.371206    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:57:10.380509 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.380484    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:10.425247 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.425217    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-c4htm-7cd77"]
Nov 02 13:57:10.446585 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.446557    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.465675 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.465648    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c4htm-7cd77"]
Nov 02 13:57:10.465915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.465881    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jk68j\""
Nov 02 13:57:10.476172 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.476148    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a62f730c-7d26-4b82-a8cc-4607533c8a5e-data-volume\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.476296 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.476215    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqdd\" (UniqueName: \"kubernetes.io/projected/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-api-access-2gqdd\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.476394 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.476317    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a62f730c-7d26-4b82-a8cc-4607533c8a5e-crio-socket\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.476457 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.476439    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.476513 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.476485    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.494170 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.494134    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"]
Nov 02 13:57:10.513076 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.513053    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:10.519283 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.519127    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 13:57:10.520964 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.520596    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"]
Nov 02 13:57:10.543629 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.543599    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 13:57:10.553084 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.553045    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:10.570496 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.570473    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5mnrf\""
Nov 02 13:57:10.577137 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577113    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a62f730c-7d26-4b82-a8cc-4607533c8a5e-data-volume\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.577282 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577152    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.577282 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577183    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.577282 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577209    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.577458 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577285    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.577458 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577325    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqdd\" (UniqueName: \"kubernetes.io/projected/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-api-access-2gqdd\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.577458 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577368    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a62f730c-7d26-4b82-a8cc-4607533c8a5e-crio-socket\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.577458 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577418    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.577458 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577452    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.577635 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577485    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a62f730c-7d26-4b82-a8cc-4607533c8a5e-data-volume\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.577635 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:10.577545    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:57:10.577635 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:10.577611    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.077596594 +0000 UTC m=+36.797725828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : configmap "kube-rbac-proxy" not found
Nov 02 13:57:10.577749 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:10.577731    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:10.578012 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:10.577803    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.077777053 +0000 UTC m=+36.797906274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:10.578012 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.577899    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a62f730c-7d26-4b82-a8cc-4607533c8a5e-crio-socket\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.634915 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.634885    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqdd\" (UniqueName: \"kubernetes.io/projected/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-api-access-2gqdd\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:10.641437 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.641405    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 13:57:10.641437 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.641424    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 13:57:10.678961 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.678801    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.678961 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.678912    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.678961 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.678943    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.679202 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.678995    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqdc\" (UniqueName: \"kubernetes.io/projected/2d33d686-5e0b-4c04-b06c-3270a104521d-kube-api-access-8xqdc\") pod \"migrator-66df5d496b-q4phj\" (UID: \"2d33d686-5e0b-4c04-b06c-3270a104521d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:10.679202 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.679032    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.679795 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.679748    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.679795 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.679773    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.680313 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.680288    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.705129 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.705093    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.771831 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.771626    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:10.779870 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.779838    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqdc\" (UniqueName: \"kubernetes.io/projected/2d33d686-5e0b-4c04-b06c-3270a104521d-kube-api-access-8xqdc\") pod \"migrator-66df5d496b-q4phj\" (UID: \"2d33d686-5e0b-4c04-b06c-3270a104521d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:10.803943 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.803863    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:10.808093 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.808036    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqdc\" (UniqueName: \"kubernetes.io/projected/2d33d686-5e0b-4c04-b06c-3270a104521d-kube-api-access-8xqdc\") pod \"migrator-66df5d496b-q4phj\" (UID: \"2d33d686-5e0b-4c04-b06c-3270a104521d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:10.840180 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.840150    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:10.953167 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.953138    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s228h"]
Nov 02 13:57:10.976724 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.975628    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:10.976724 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.976667    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s228h"]
Nov 02 13:57:10.991121 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.991095    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 13:57:10.991121 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.991106    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 13:57:10.991277 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.991171    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 13:57:10.991277 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.991207    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 13:57:10.991277 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:10.991253    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lfskl\""
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.083000    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.083053    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-cabundle\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.083144    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-key\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.083162    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jv62\" (UniqueName: \"kubernetes.io/projected/dbf753aa-a9de-4930-8d1e-546dcbf74281-kube-api-access-4jv62\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.083191    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:11.083290    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:11.083345    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:12.083333036 +0000 UTC m=+37.803462255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:11.083685    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:57:11.083823 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:11.083729    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:12.08371734 +0000 UTC m=+37.803846560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : configmap "kube-rbac-proxy" not found
Nov 02 13:57:11.097487 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.097457    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c4htm-7cd77"]
Nov 02 13:57:11.102816 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:57:11.102769    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33196c6d_9ade_4612_99bd_4e757677ec89.slice/crio-5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5 WatchSource:0}: Error finding container 5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5: Status 404 returned error can't find the container with id 5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5
Nov 02 13:57:11.153163 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.153121    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c4htm-7cd77" event={"ID":"33196c6d-9ade-4612-99bd-4e757677ec89","Type":"ContainerStarted","Data":"5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5"}
Nov 02 13:57:11.173103 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.173068    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"]
Nov 02 13:57:11.178935 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:57:11.178907    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d33d686_5e0b_4c04_b06c_3270a104521d.slice/crio-9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03 WatchSource:0}: Error finding container 9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03: Status 404 returned error can't find the container with id 9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03
Nov 02 13:57:11.184088 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.184060    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-key\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.184191 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.184114    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv62\" (UniqueName: \"kubernetes.io/projected/dbf753aa-a9de-4930-8d1e-546dcbf74281-kube-api-access-4jv62\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.184247 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.184188    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-cabundle\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.185086 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.185057    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-cabundle\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.188230 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.188210    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-key\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.224024 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.223993    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv62\" (UniqueName: \"kubernetes.io/projected/dbf753aa-a9de-4930-8d1e-546dcbf74281-kube-api-access-4jv62\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.300194 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.300156    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:11.406555 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.406484    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:11.520382 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.520280    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:11.641529 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:11.641438    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s228h"]
Nov 02 13:57:11.647513 ip-10-0-129-115 kubenswrapper[2542]: W1102 13:57:11.647485    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf753aa_a9de_4930_8d1e_546dcbf74281.slice/crio-c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe WatchSource:0}: Error finding container c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe: Status 404 returned error can't find the container with id c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe
Nov 02 13:57:12.041735 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:12.041659    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:12.091879 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:12.091839    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:12.092049 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:12.091894    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:12.092049 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:12.092002    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:57:12.092049 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:12.092048    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:14.092035039 +0000 UTC m=+39.812164262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : configmap "kube-rbac-proxy" not found
Nov 02 13:57:12.092476 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:12.092456    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:12.092622 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:12.092597    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:14.092562404 +0000 UTC m=+39.812691627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:12.157860 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:12.157821    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj" event={"ID":"2d33d686-5e0b-4c04-b06c-3270a104521d","Type":"ContainerStarted","Data":"9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03"}
Nov 02 13:57:12.159422 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:12.159394    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s228h" event={"ID":"dbf753aa-a9de-4930-8d1e-546dcbf74281","Type":"ContainerStarted","Data":"c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe"}
Nov 02 13:57:13.001661 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:13.001022    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:13.001661 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:13.001204    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:13.001661 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:13.001286    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:21.001246031 +0000 UTC m=+46.721375282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:14.117725 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.117684    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:14.118176 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.117738    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:14.118176 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:14.117855    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:57:14.118176 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:14.117938    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.117918673 +0000 UTC m=+43.838047973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : configmap "kube-rbac-proxy" not found
Nov 02 13:57:14.118176 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:14.117859    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:14.118176 ip-10-0-129-115 kubenswrapper[2542]: E1102 13:57:14.118023    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.118004154 +0000 UTC m=+43.838133389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:14.167110 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.167073    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8cskr" event={"ID":"8e854bb3-3994-496f-bc04-3ace19a132a9","Type":"ContainerStarted","Data":"8c57e7e5cf2e94fb00709c6f343e5f792cabe4cfe4913b1f6733ad935995278c"}
Nov 02 13:57:14.167287 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.167195    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:14.169000 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.168948    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj" event={"ID":"2d33d686-5e0b-4c04-b06c-3270a104521d","Type":"ContainerStarted","Data":"60263539a0d7e4c091fcb91df96cc1682537fa4f3a6bbd971e2a56092dce2b12"}
Nov 02 13:57:14.207140 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.207089    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8cskr" podStartSLOduration=34.960432174 podStartE2EDuration="39.207072135s" totalImagesPullingTime="4.246639961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:14.206332356 +0000 UTC m=+39.926461600" watchObservedRunningTime="2026-11-02 13:57:14.207072135 +0000 UTC m=+39.927201376"
Nov 02 13:57:14.249162 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.249106    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj" podStartSLOduration=2.362568052 podStartE2EDuration="4.249088656s" totalImagesPullingTime="1.886520604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:14.248450585 +0000 UTC m=+39.968579827" watchObservedRunningTime="2026-11-02 13:57:14.249088656 +0000 UTC m=+39.969217899"
Nov 02 13:57:14.449332 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.449285    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:14.587892 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.587859    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:14.701929 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:14.701859    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:15.172417 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:15.172384    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c4htm-7cd77" event={"ID":"33196c6d-9ade-4612-99bd-4e757677ec89","Type":"ContainerStarted","Data":"fe6b3e95565d55f7b634a335d16a77aff6aecf4f0c9f0ea927d65944ddff3caf"}
Nov 02 13:57:15.216054 ip-10-0-129-115 kubenswrapper[2542]: I1102 13:57:15.216004    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-c4htm-7cd77" podStartSLOduration=2.085400794 podStartE2EDuration="5.215990507s" totalImagesPullingTime="3.130589713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:15.215175404 +0000 UTC m=+40.935304647" watchObservedRunningTime="2026-11-02 13:57:15.215990507 +0000 UTC m=+40.936119750"
Nov 02 13:57:15.393573 ip-10-0-129-115 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 13:57:15.409752 ip-10-0-129-115 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 13:57:15.409931 ip-10-0-129-115 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 13:57:15.409985 ip-10-0-129-115 systemd[1]: kubelet.service: Consumed 2.526s CPU time.
Nov 02 13:57:15.425875 ip-10-0-129-115 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:57:15.487295 ip-10-0-129-115 kubenswrapper[4492]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:57:15.487295 ip-10-0-129-115 kubenswrapper[4492]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:57:15.487295 ip-10-0-129-115 kubenswrapper[4492]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:57:15.487295 ip-10-0-129-115 kubenswrapper[4492]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:57:15.489327 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489246    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.489327 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489327    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.489327 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489331    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489334    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489337    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489340    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489342    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489345    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489347    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489363    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489366    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489369    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489371    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489374    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489377    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489379    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489382    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489384    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489387    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489389    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489392    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489394    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.489486 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489397    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489399    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489402    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489404    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489407    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489410    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489412    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489418    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489422    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489425    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489428    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489431    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489434    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489436    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489439    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489441    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489444    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489447    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489449    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.489991 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489452    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489455    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489457    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489459    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489462    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489465    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489467    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489470    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489474    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489478    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489483    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489486    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489489    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489492    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489494    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489498    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489500    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489503    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489505    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.490530 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489508    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489510    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489513    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489515    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489518    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489520    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489523    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489526    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489528    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489531    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489533    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489536    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489538    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489540    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489543    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489545    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489548    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489550    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489554    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489557    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.491071 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489560    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489562    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489565    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489567    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489570    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489572    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489574    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489577    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489579    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489582    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489584    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489587    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489589    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489591    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489594    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489596    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489599    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489601    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489605    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.491587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489610    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489614    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489616    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489619    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489622    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489624    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.489627    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490460    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490468    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490471    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490474    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490476    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490479    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490510    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490514    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490517    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490520    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490523    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490525    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.492049 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490528    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490531    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490533    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490536    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490539    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490541    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490544    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490561    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490564    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490567    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490570    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490572    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490576    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490579    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490581    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490584    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490586    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490589    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490592    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.492551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490595    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490598    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490600    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490603    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490606    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490608    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490611    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490615    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490617    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490620    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490622    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490625    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490627    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490630    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490632    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490635    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490638    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490641    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490643    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490646    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.493023 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490648    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490651    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490653    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490656    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490658    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490661    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490663    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490665    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490668    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490670    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490673    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490675    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490678    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490680    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490682    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490685    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490688    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490690    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490693    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490696    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.493552 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490699    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490701    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490708    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490711    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490714    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490716    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490719    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490721    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490725    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490729    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490732    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490736    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490739    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490741    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490744    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490747    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490749    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490752    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490755    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.494054 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490757    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490760    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490762    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490765    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490767    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490770    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490775    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490779    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490782    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490789    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490792    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490795    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490798    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490800    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490804    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.490807    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491296    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491301    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491304    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.494547 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491307    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491309    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491312    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491315    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491317    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491320    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491323    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491325    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491328    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491332    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491336    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491338    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491341    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491344    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491347    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491349    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491371    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491375    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491377    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491380    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.495065 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491382    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491385    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491388    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491390    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491393    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491395    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491398    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491401    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491404    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491407    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491409    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491412    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491415    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491417    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491419    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491422    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491424    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491427    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491429    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.495618 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491431    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491434    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491436    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491439    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491441    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491443    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491446    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491448    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491451    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491453    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491459    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491461    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491464    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491466    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491468    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491471    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491473    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491476    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491478    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491481    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.496091 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491483    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491486    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491489    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491492    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491495    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491497    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491500    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491502    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491505    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491507    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491510    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491513    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491515    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491517    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491520    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491522    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491524    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491527    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491529    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.496605 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491532    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491534    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491536    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491540    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491543    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491545    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491548    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491550    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491555    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491558    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491560    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491563    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491565    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491568    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491570    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491573    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491576    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491578    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491581    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.497138 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491585    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491589    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491592    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491596    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491599    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.491601    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491692    4492 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491700    4492 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491709    4492 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491716    4492 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491722    4492 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491727    4492 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491732    4492 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491736    4492 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491739    4492 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491742    4492 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491748    4492 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491751    4492 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491754    4492 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491757    4492 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491760    4492 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491768    4492 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491770    4492 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:57:15.497622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491773    4492 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491778    4492 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491781    4492 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491784    4492 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491787    4492 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491790    4492 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491795    4492 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491798    4492 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491801    4492 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491804    4492 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491807    4492 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491810    4492 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491813    4492 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491816    4492 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491819    4492 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491823    4492 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491826    4492 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491828    4492 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491831    4492 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491834    4492 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491837    4492 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491841    4492 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491844    4492 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491847    4492 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:57:15.498322 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491850    4492 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491855    4492 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491859    4492 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491862    4492 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491865    4492 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491868    4492 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491872    4492 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491875    4492 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491878    4492 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491880    4492 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491883    4492 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491886    4492 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491889    4492 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491898    4492 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491901    4492 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491905    4492 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491908    4492 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491911    4492 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491914    4492 flags.go:64] FLAG: --help="false"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491917    4492 flags.go:64] FLAG: --hostname-override="ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491920    4492 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491923    4492 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491926    4492 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491929    4492 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:57:15.498969 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491932    4492 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491935    4492 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491938    4492 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491941    4492 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491944    4492 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491946    4492 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491949    4492 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491952    4492 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491955    4492 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491959    4492 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491962    4492 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491965    4492 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491968    4492 flags.go:64] FLAG: --lock-file=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491970    4492 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491974    4492 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491977    4492 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491983    4492 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491986    4492 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491989    4492 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491992    4492 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491995    4492 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.491998    4492 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492001    4492 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492004    4492 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492010    4492 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:57:15.499590 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492013    4492 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492016    4492 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492019    4492 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492022    4492 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492025    4492 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492028    4492 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492031    4492 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492033    4492 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492036    4492 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492043    4492 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492046    4492 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492049    4492 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492052    4492 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492055    4492 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492058    4492 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492061    4492 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492065    4492 flags.go:64] FLAG: --port="10250"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492068    4492 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492071    4492 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a897f8dc18ece89"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492075    4492 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492078    4492 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492082    4492 flags.go:64] FLAG: --register-node="true"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492085    4492 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492088    4492 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492092    4492 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492095    4492 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:57:15.500270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492098    4492 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492101    4492 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492104    4492 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492107    4492 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492111    4492 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492113    4492 flags.go:64] FLAG: --runonce="false"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492119    4492 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492123    4492 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492126    4492 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492128    4492 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492131    4492 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492134    4492 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492137    4492 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492140    4492 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492143    4492 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492146    4492 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492149    4492 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492152    4492 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492155    4492 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492158    4492 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492160    4492 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492165    4492 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492170    4492 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492173    4492 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492177    4492 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:57:15.500939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492180    4492 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492183    4492 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492187    4492 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492190    4492 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492193    4492 flags.go:64] FLAG: --v="2"
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492197    4492 flags.go:64] FLAG: --version="false"
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492201    4492 flags.go:64] FLAG: --vmodule=""
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492205    4492 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492208    4492 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492340    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492344    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492346    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492349    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492367    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492372    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492375    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492379    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492383    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492386    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492389    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492391    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.501669 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492396    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492400    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492404    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492406    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492409    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492412    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492414    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492417    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492421    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492424    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492426    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492429    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492432    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492435    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492438    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492441    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492443    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492446    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492448    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492451    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.502237 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492454    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492456    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492459    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492461    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492464    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492466    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492469    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492472    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492474    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492477    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492479    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492482    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492484    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492486    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492489    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492493    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492496    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492499    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492502    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492505    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.503058 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492510    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492512    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492516    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492518    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492521    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492524    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492527    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492530    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492532    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492535    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492537    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492540    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492542    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492545    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492547    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492550    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492553    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492555    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492558    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.503859 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492561    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492564    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492567    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492569    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492571    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492574    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492576    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492578    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492581    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492583    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492586    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492588    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492590    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492594    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492597    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492599    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492602    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492604    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492608    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492611    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.504580 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492613    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492616    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492618    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492621    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492624    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492626    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492628    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492631    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492634    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492636    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492638    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492641    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492643    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492646    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492649    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.492675    4492 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:57:15.505156 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492711    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492715    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492719    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492722    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492725    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492727    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492730    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492733    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492735    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492737    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492741    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492744    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492747    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492749    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492753    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492756    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492759    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492761    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492763    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492766    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.505728 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492769    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492771    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492775    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492779    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492781    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492784    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492787    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492790    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492792    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492795    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492798    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492800    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492802    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492805    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492807    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492810    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492812    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492815    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492817    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.506571 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492819    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492822    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492824    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492828    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492831    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492842    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492845    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492849    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492852    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492854    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492857    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492860    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492862    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492865    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492867    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492869    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492872    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492874    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492877    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.507242 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492879    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492882    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492884    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492888    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492891    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492894    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492897    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492899    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492902    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492905    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492907    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492909    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492912    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492914    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492917    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492919    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492923    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492925    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492928    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492930    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.507889 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492934    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492936    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492939    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492941    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492944    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492946    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492949    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492951    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492954    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492956    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492959    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492961    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492963    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492966    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492968    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492972    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492976    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492979    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492982    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.508508 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492985    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492988    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492990    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492992    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.492995    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.493003    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.493005    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.493008    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.493011    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.497547    4492 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.497566    4492 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.497588    4492 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.497596    4492 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497675    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497683    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497689    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497693    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497698    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497702    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.509124 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497706    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497710    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497714    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497718    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497722    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497727    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497731    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497735    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497739    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497743    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497747    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497751    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497755    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497759    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497763    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497767    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497771    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497776    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497779    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497784    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.509795 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497787    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497792    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497796    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497808    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497812    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497816    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497820    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497824    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497828    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497832    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497836    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497840    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497844    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497848    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497851    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497855    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497859    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497863    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497867    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.510408 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497872    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497876    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497880    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497884    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497888    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497892    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497896    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497900    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497904    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497908    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497912    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497916    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497919    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497923    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497927    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497931    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497935    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497940    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497944    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497947    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.510910 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497951    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497955    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497959    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497962    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497967    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497971    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497975    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497978    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497982    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497986    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497990    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497993    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.497998    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498004    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498010    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498014    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498018    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498021    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498025    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.511499 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498030    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498034    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498037    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498041    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498045    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498049    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498053    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498057    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498060    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498064    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498068    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498073    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498077    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498083    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498088    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498093    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498097    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498101    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498105    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.511960 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498109    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498113    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498117    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.498157    4492 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498380    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498387    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498391    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498395    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498399    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498403    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498408    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498412    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498416    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498420    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498424    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498429    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.512482 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498435    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498441    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498446    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498450    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498455    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498459    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498464    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498468    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498471    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498476    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498480    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498484    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498488    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498492    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498496    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498500    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498504    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498508    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498512    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498516    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.512874 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498520    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498524    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498528    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498532    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498536    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498540    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498544    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498548    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498552    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498556    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498560    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498564    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498568    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498572    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498578    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498584    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498588    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498592    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498596    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.513418 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498600    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498604    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498608    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498613    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498617    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498621    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498624    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498628    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498632    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498636    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498640    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498644    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498648    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498653    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498657    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498661    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498665    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498669    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498673    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.514069 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498676    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498680    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498685    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498689    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498693    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498697    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498702    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498706    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498710    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498714    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498718    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498721    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498725    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498729    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498733    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498737    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498741    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498745    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498749    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498753    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.514835 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498757    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498761    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498765    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498769    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498772    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498777    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498781    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498785    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498789    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498793    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498797    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498801    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498805    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498810    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498814    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498818    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.515323 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.498853    4492 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498901    4492 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498906    4492 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498910    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498914    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498918    4492 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498924    4492 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498930    4492 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498934    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498938    4492 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498942    4492 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498946    4492 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498950    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498953    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498958    4492 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498962    4492 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498966    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498970    4492 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498974    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:57:15.515791 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498979    4492 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498983    4492 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498987    4492 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498991    4492 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498995    4492 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.498999    4492 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499002    4492 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499006    4492 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499010    4492 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499014    4492 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499018    4492 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499022    4492 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499026    4492 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499030    4492 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499033    4492 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499038    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499041    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499045    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499049    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499053    4492 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:57:15.516296 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499057    4492 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499064    4492 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499069    4492 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499073    4492 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499077    4492 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499081    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499085    4492 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499089    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499093    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499097    4492 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499101    4492 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499105    4492 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499109    4492 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499113    4492 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499118    4492 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499122    4492 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499126    4492 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499129    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499133    4492 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:57:15.516798 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499137    4492 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499141    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499145    4492 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499149    4492 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499153    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499157    4492 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499161    4492 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499165    4492 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499169    4492 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499173    4492 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499177    4492 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499181    4492 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499185    4492 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499193    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499197    4492 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499201    4492 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499205    4492 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499208    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499213    4492 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499216    4492 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:57:15.517263 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499220    4492 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499224    4492 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499229    4492 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499233    4492 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499236    4492 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499240    4492 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499245    4492 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499249    4492 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499253    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499256    4492 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499261    4492 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499265    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499268    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499272    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499276    4492 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499280    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499284    4492 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499288    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499292    4492 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499296    4492 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:57:15.517781 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499300    4492 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499304    4492 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499309    4492 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499313    4492 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499317    4492 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499323    4492 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499327    4492 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499331    4492 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:15.499335    4492 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.499435    4492 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.500887    4492 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.500949    4492 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.501203    4492 server.go:1014] "Starting client certificate rotation"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.501338    4492 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.501405    4492 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:51:35 +0000 UTC" deadline="2028-05-23 02:20:51.464689475 +0000 UTC"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.501458    4492 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13620h23m35.963234855s"
Nov 02 13:57:15.518326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.501879    4492 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:57:15.518965 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.505120    4492 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:57:15.518965 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.506979    4492 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:57:15.518965 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.508844    4492 log.go:25] "Validated CRI v1 image API"
Nov 02 13:57:15.518965 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.509617    4492 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:57:15.518965 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.511230    4492 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 94720379-0f7f-4a75-a6e8-b841fc761c57:/dev/nvme0n1p4 b6930021-2899-4cc4-83be-7668c49c215d:/dev/nvme0n1p3]
Nov 02 13:57:15.519200 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.511250    4492 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b/volumes/kubernetes.io~projected/kube-api-access-brhbn:{mountpoint:/var/lib/kubelet/pods/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b/volumes/kubernetes.io~projected/kube-api-access-brhbn major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d33d686-5e0b-4c04-b06c-3270a104521d/volumes/kubernetes.io~projected/kube-api-access-8xqdc:{mountpoint:/var/lib/kubelet/pods/2d33d686-5e0b-4c04-b06c-3270a104521d/volumes/kubernetes.io~projected/kube-api-access-8xqdc major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33196c6d-9ade-4612-99bd-4e757677ec89/volumes/kubernetes.io~projected/kube-api-access-c42cj:{mountpoint:/var/lib/kubelet/pods/33196c6d-9ade-4612-99bd-4e757677ec89/volumes/kubernetes.io~projected/kube-api-access-c42cj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3/volumes/kubernetes.io~projected/kube-api-access-xdlxx:{mountpoint:/var/lib/kubelet/pods/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3/volumes/kubernetes.io~projected/kube-api-access-xdlxx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/519bdf1e-bdde-4785-a71c-74c0d80ea527/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/519bdf1e-bdde-4785-a71c-74c0d80ea527/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d5593ba-0e81-4d12-afbc-73019f33d7dd/volumes/kubernetes.io~projected/kube-api-access-n479m:{mountpoint:/var/lib/kubelet/pods/6d5593ba-0e81-4d12-afbc-73019f33d7dd/volumes/kubernetes.io~projected/kube-api-access-n479m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e9884e1-a236-4925-9875-53f6b1e64ba0/volumes/kubernetes.io~projected/kube-api-access-m9hdn:{mountpoint:/var/lib/kubelet/pods/6e9884e1-a236-4925-9875-53f6b1e64ba0/volumes/kubernetes.io~projected/kube-api-access-m9hdn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f928dc3-3c2e-4baa-b29b-de0a84eed156/volumes/kubernetes.io~projected/kube-api-access-vfzfl:{mountpoint:/var/lib/kubelet/pods/7f928dc3-3c2e-4baa-b29b-de0a84eed156/volumes/kubernetes.io~projected/kube-api-access-vfzfl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e854bb3-3994-496f-bc04-3ace19a132a9/volumes/kubernetes.io~projected/kube-api-access-t5ggh:{mountpoint:/var/lib/kubelet/pods/8e854bb3-3994-496f-bc04-3ace19a132a9/volumes/kubernetes.io~projected/kube-api-access-t5ggh major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62f730c-7d26-4b82-a8cc-4607533c8a5e/volumes/kubernetes.io~projected/kube-api-access-2gqdd:{mountpoint:/var/lib/kubelet/pods/a62f730c-7d26-4b82-a8cc-4607533c8a5e/volumes/kubernetes.io~projected/kube-api-access-2gqdd major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volumes/kubernetes.io~projected/kube-api-access-hj42k:{mountpoint:/var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volumes/kubernetes.io~projected/kube-api-access-hj42k major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284/volumes/kubernetes.io~projected/kube-api-access-264tr:{mountpoint:/var/lib/kubelet/pods/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284/volumes/kubernetes.io~projected/kube-api-access-264tr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~projected/kube-api-access-8ccx5:{mountpoint:/var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~projected/kube-api-access-8ccx5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbf753aa-a9de-4930-8d1e-546dcbf74281/volumes/kubernetes.io~projected/kube-api-access-4jv62:{mountpoint:/var/lib/kubelet/pods/dbf753aa-a9de-4930-8d1e-546dcbf74281/volumes/kubernetes.io~projected/kube-api-access-4jv62 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbf753aa-a9de-4930-8d1e-546dcbf74281/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/dbf753aa-a9de-4930-8d1e-546dcbf74281/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd07b61c-b867-47ee-ae89-d324cbcaf032/volumes/kubernetes.io~projected/kube-api-access-5jpdb:{mountpoint:/var/lib/kubelet/pods/dd07b61c-b867-47ee-ae89-d324cbcaf032/volumes/kubernetes.io~projected/kube-api-access-5jpdb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de9d0ea2-cd91-46df-b5fb-f322312c3390/volumes/kubernetes.io~projected/kube-api-access-5jfpn:{mountpoint:/var/lib/kubelet/pods/de9d0ea2-cd91-46df-b5fb-f322312c3390/volumes/kubernetes.io~projected/kube-api-access-5jfpn major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de9d0ea2-cd91-46df-b5fb-f322312c3390/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/de9d0ea2-cd91-46df-b5fb-f322312c3390/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3/volumes/kubernetes.io~projected/kube-api-access-5n5mh:{mountpoint:/var/lib/kubelet/pods/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3/volumes/kubernetes.io~projected/kube-api-access-5n5mh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9047a53-882d-4708-945a-663dcdc7dc9a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f9047a53-882d-4708-945a-663dcdc7dc9a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5d25a3a5d2ac8c24c0604151a882b46bff8d495fbeb07bfb2e1c83e4d9063904/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a7f2dae22533c1496b37d4697053a849da0a825143888171c08c51534284d3aa/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/10e881816b5d5f58b5ae11a094be4b37588a7466c9af8935926eef0cde6ed313/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8c652c5afa6dcf194b6e8d0a4e771c40843baad13056c85ee3e08a38a14032d1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5a3b7844d21004ba568c149479439806d7221b891b9f569c1e851aadef744b52/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7ed216f15bb9b98d6f3108958327ae37f5ef9fac4ebf173675d2f35914cefa2e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/359c491598695b74725ea35755bf3b08291c7b325867882924ae2b4c3c2d413f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7ccd0bed1e565e3e78304c34b12a9ec58dfae044d1a1a78d7e2c218c41c47c86/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b125721e2731836898b80cf98d6c9e100f3754b4d38050cef0de42b6e9666f4b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/8327a3e2c7de652d8ddbb130c442d79e2f51455753c1bee313d8a33d1ed32f86/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/6e1d2d5460afb387ffb78b71022280083f317a320fa4d0565ecabca557b5aa1a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ae4126bfc04fdf00cb8eafdd4c49de5195fd3b11beb42446a184cb21951ed60c/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/207dc94098ac2365a6436afe63c29403a63b2a779f09889d201c87cd07745366/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cf9a8ba260fc3a084b00b334a1352016ecfa4c1ab99b86468b7f3fb2fdae73fd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/52c7ba71ae36233b8fbb8a759aa21ce5d9312bbbeac2a62f783ceedb4a6f2506/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f0fedb4acd7b6cc66ad9ac87d6c4873b394a8b2bc574dfaa8fcb9fbde4b44e99/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/93a6f0cb43a92f694ffadb21631a8b133b6e0a462d2871aa2f56449a6add4d62/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3b07c61b6226c56f79099a3f375b9ea38f9fcccad8d87df81d04a0d843c0475d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0f37ed097ec1bfb8ddf2a396b3b8c0630984dbaeeaac48831254e7ad3ecd5c09/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8ce013af02935fbccbc20706be31addc12fc70d4e0383826e62ec1f3c6b91176/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c49496162563330d39934810acde20a8ad17ed89d9004867d28265eace45b916/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/5d3a634c003b18f8fdff47cb2cd880a424262fbc30b9803b6a8a3a3d0140a490/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/3e7163f2dc5e95dc7d889a20d007d46db044c45d0354d973c6d9e18970f5449c/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/486e5edfe7836fbedd14ad6f959974283ba97939e1e0c19218a73a643156f811/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/84707242a12ce113794cd528035e3cd696d46296d7e2de74a73c5879786972c0/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/b57950ebc010193d32437f6f5c040ac4bbf98592c0fd8fbf9e9c79c10e250212/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/b20d2721b0013a1eb82291cbece7725d6e147016bac7db886b19979b1568c921/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/ee44f226dceddee206ab9faab740b34a6400824a59a43658ab1a3e4cace2dc9a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/8e149d4b8c6bfb62b388c295215fcc882e28d165d3de51d56e8bf95eebedfbc9/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/0f2214ecfc0441807af2358ffd2fd7205b370e3ff5c33ab4be03d3a2d51054db/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/04418a5d3416fef5e4bf3e7a99a116893041b3329f135eaddd74a14a0b186c60/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/865e723b2ebd5c036ac51a7fc6ce2c0ea8b6f00786219e7d5ed8a607b7308403/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/403dc6ab042aef73497f012fbe5eb789afc47f2d1ae57ff11f6f2d9893734d3a/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/ec7beb9025c3e6d75b8fa04b1736f21052bfd65657820dc26d465455adb17d6f/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/15d0c2172c2b9a49935d4388880cd17bfe0014d5a356ec9fa08148d1c24b8687/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9683223a6b78f1a63c8769d122a7065083d53c36c442d45f454ba0ed129d5591/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/985eeb30710109c84a4c8b1f6393fab487cc589cea9b78c51e55989a6162bc81/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b5e622d32cb4b5f24a4e4098ee412ed8d79f24b6298ce72365429b6fa45b87ad/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/cecb81774cc83b3952b1634fe8148897cb0a605c8fda0d597d47d6fd751c6446/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/f3b04f0bba0fe603c2299edc9215d428f84bdf18807f3d92f93a02294af65f65/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/7cb686ba447428af3de9f49608fb1cb0aaa5590cd49dd987918644611a1e353b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/10460c5b153e74d10ae0cae8db8a6af69b8c9d8f6055e1738091d846a355154e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b4eca91a0ee996967e9cb22d077ca76be60bcbdf629e212dd9d7aea723480f0c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.518087    4492 manager.go:217] Machine: {Timestamp:2026-11-02 13:57:15.517618032 +0000 UTC m=+0.061333618 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102402 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b74a1f06bea11e0e3162eea19f8a5 SystemUUID:ec2b74a1-f06b-ea11-e0e3-162eea19f8a5 BootID:49a54c6c-d663-49df-b46a-5975d2732a4a Filesystems:[{Device:/run/containers/storage/overlay-containers/82759dec6f0ca968a8ec1035e3ddcfd9e95f0ab187fd503254a488b940a5646d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06aef6c36449e41c7f95bcde3cc7e97af56a7645d32eea9c5770579c62358830/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a149da9bec43b22ded257237b8a3bbf6321b1707292b1037706e9f04b4a5ef4/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3/volumes/kubernetes.io~projected/kube-api-access-5n5mh DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14bfaf58cb7e948f17e81a46fe3ccd1eb0cf5d14d66bfcc5a46188b30259d564/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33196c6d-9ade-4612-99bd-4e757677ec89/volumes/kubernetes.io~projected/kube-api-access-c42cj DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dbf753aa-a9de-4930-8d1e-546dcbf74281/volumes/kubernetes.io~projected/kube-api-access-4jv62 DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/519bdf1e-bdde-4785-a71c-74c0d80ea527/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f928dc3-3c2e-4baa-b29b-de0a84eed156/volumes/kubernetes.io~projected/kube-api-access-vfzfl DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~projected/kube-api-access-8ccx5 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b/volumes/kubernetes.io~projected/kube-api-access-brhbn DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec3e431948a7928480b9312d6106533610cffbd0d766596b252cf078d4c259c9/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67eb896e132451fd1f0e8a50d13e465014578d9ae923ecbdcc7ec24ba3aa9648/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6d5593ba-0e81-4d12-afbc-73019f33d7dd/volumes/kubernetes.io~projected/kube-api-access-n479m DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fc4a7379b93e6e14476a8cfc2aebd2728f5f4c49c3f3a83d305868c49d9a0a7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cec63cc5-8483-4e4f-aca4-2160dab7a670/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e854bb3-3994-496f-bc04-3ace19a132a9/volumes/kubernetes.io~projected/kube-api-access-t5ggh DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dbf753aa-a9de-4930-8d1e-546dcbf74281/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aac9e728bcc103f5a23a381d49e24901be46009ae2cfc0d8eb4f240bfd070bed/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8ba699aeba4c5eed359679b19202189ef646bca07eaca844c34c99310910a97/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de9d0ea2-cd91-46df-b5fb-f322312c3390/volumes/kubernetes.io~projected/kube-api-access-5jfpn DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a62f730c-7d26-4b82-a8cc-4607533c8a5e/volumes/kubernetes.io~projected/kube-api-access-2gqdd DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2d33d686-5e0b-4c04-b06c-3270a104521d/volumes/kubernetes.io~projected/kube-api-access-8xqdc DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6e9884e1-a236-4925-9875-53f6b1e64ba0/volumes/kubernetes.io~projected/kube-api-access-m9hdn DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de9d0ea2-cd91-46df-b5fb-f322312c3390/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2357541cf2de36167ae4e2d8d9c4ba63c9ff2e2218f3aa830bf93ab3d54a2808/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3/volumes/kubernetes.io~projected/kube-api-access-xdlxx DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284/volumes/kubernetes.io~projected/kube-api-access-264tr DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volumes/kubernetes.io~projected/kube-api-access-hj42k DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ceddae9f7b082693da656c1d410a8e90333496e06cd23e01e85fec42dd45226d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b19899c7-dc0f-439b-aa44-5948528cd810/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9417e6077d221e80c3073fa5ed6aa854c9dcc3127b40f6943d6f2c0134617a03/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5111c989454386e42ca7fa0a65124c60c6559b2557607ce20a76f7f91c467fe/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9047a53-882d-4708-945a-663dcdc7dc9a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd07b61c-b867-47ee-ae89-d324cbcaf032/volumes/kubernetes.io~projected/kube-api-access-5jpdb DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2357541cf2de361 MacAddress:46:c1:6d:25:43:6f Speed:10000 Mtu:8901} {Name:5e4dfec36a79738 MacAddress:a2:70:2a:b6:75:36 Speed:10000 Mtu:8901} {Name:6a149da9bec43b2 MacAddress:76:a6:da:71:7d:4e Speed:10000 Mtu:8901} {Name:9417e6077d221e8 MacAddress:ae:b5:6a:dd:c8:ec Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:eb:5e:60:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:dc:ed:13:3a:87 Speed:0 Mtu:8901} {Name:c5111c989454386 MacAddress:3e:c0:e6:7b:da:2d Speed:10000 Mtu:8901} {Name:ec3e431948a7928 MacAddress:c2:4f:1b:6b:ea:d4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:eb:5e:60:f9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:36:9e:21:f6:ef:0d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:90:45:71:11:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.518348    4492 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.518410    4492 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.518827    4492 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.518854    4492 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519072    4492 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519085    4492 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519104    4492 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519131    4492 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519254    4492 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:57:15.519528 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519524    4492 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:57:15.520134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519681    4492 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:57:15.520134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519729    4492 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:57:15.520134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519746    4492 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:57:15.520134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519759    4492 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:57:15.520134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519772    4492 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:57:15.520134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.519783    4492 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:57:15.520704 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.520686    4492 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:57:15.520704 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.520702    4492 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:57:15.521487 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.521471    4492 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:57:15.534150 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534128    4492 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:57:15.534150 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534150    4492 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:57:15.534340 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534300    4492 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 13:57:15.534652 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534634    4492 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:57:15.534736 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534684    4492 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:57:15.534821 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534809    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:57:15.534885 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534831    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:57:15.534885 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534841    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:57:15.534885 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534850    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:57:15.534885 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534860    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:57:15.534885 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534874    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:57:15.534885 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534884    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534893    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534903    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534913    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534926    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534941    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534976    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:57:15.535213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.534985    4492 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:57:15.539698 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.539680    4492 server.go:1263] "Started kubelet"
Nov 02 13:57:15.539781 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.539728    4492 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:57:15.540134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.540029    4492 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:57:15.540134 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.540120    4492 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:57:15.540302 ip-10-0-129-115 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:57:15.540622 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.540600    4492 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:57:15.545656 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.545626    4492 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548104    4492 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548122    4492 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548160    4492 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-06-28 14:23:37.162823934 +0000 UTC"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548194    4492 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14496h26m21.614632879s"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548225    4492 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:15.548226    4492 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-115.ec2.internal\" not found"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548232    4492 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:57:15.549225 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.548241    4492 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:57:15.549865 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.549846    4492 factory.go:55] Registering systemd factory
Nov 02 13:57:15.549865 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.549866    4492 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:57:15.550103 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550064    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b" volumeName="kubernetes.io/empty-dir/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-tmp-dir" seLinuxMountContext=""
Nov 02 13:57:15.550167 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550107    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec63cc5-8483-4e4f-aca4-2160dab7a670" volumeName="kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-tmp" seLinuxMountContext=""
Nov 02 13:57:15.550167 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550123    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbf753aa-a9de-4930-8d1e-546dcbf74281" volumeName="kubernetes.io/configmap/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-cabundle" seLinuxMountContext=""
Nov 02 13:57:15.550167 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550135    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd07b61c-b867-47ee-ae89-d324cbcaf032" volumeName="kubernetes.io/projected/dd07b61c-b867-47ee-ae89-d324cbcaf032-kube-api-access-5jpdb" seLinuxMountContext=""
Nov 02 13:57:15.550167 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550147    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d33d686-5e0b-4c04-b06c-3270a104521d" volumeName="kubernetes.io/projected/2d33d686-5e0b-4c04-b06c-3270a104521d-kube-api-access-8xqdc" seLinuxMountContext=""
Nov 02 13:57:15.550167 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550162    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3" volumeName="kubernetes.io/empty-dir/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-tmp-dir" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550174    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62f730c-7d26-4b82-a8cc-4607533c8a5e" volumeName="kubernetes.io/empty-dir/a62f730c-7d26-4b82-a8cc-4607533c8a5e-data-volume" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550185    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbf753aa-a9de-4930-8d1e-546dcbf74281" volumeName="kubernetes.io/projected/dbf753aa-a9de-4930-8d1e-546dcbf74281-kube-api-access-4jv62" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550200    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd07b61c-b867-47ee-ae89-d324cbcaf032" volumeName="kubernetes.io/configmap/dd07b61c-b867-47ee-ae89-d324cbcaf032-serviceca" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550213    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f25f0e9a-bcc1-4ce1-9361-b98df140e5f3" volumeName="kubernetes.io/projected/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kube-api-access-5n5mh" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550225    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33196c6d-9ade-4612-99bd-4e757677ec89" volumeName="kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550237    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5593ba-0e81-4d12-afbc-73019f33d7dd" volumeName="kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550249    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5593ba-0e81-4d12-afbc-73019f33d7dd" volumeName="kubernetes.io/projected/6d5593ba-0e81-4d12-afbc-73019f33d7dd-kube-api-access-n479m" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550262    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e9884e1-a236-4925-9875-53f6b1e64ba0" volumeName="kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550278    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e854bb3-3994-496f-bc04-3ace19a132a9" volumeName="kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550298    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62f730c-7d26-4b82-a8cc-4607533c8a5e" volumeName="kubernetes.io/projected/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-api-access-2gqdd" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550314    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec63cc5-8483-4e4f-aca4-2160dab7a670" volumeName="kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-tuned" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550328    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cec63cc5-8483-4e4f-aca4-2160dab7a670" volumeName="kubernetes.io/projected/cec63cc5-8483-4e4f-aca4-2160dab7a670-kube-api-access-8ccx5" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550340    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33196c6d-9ade-4612-99bd-4e757677ec89" volumeName="kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550368    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de9d0ea2-cd91-46df-b5fb-f322312c3390" volumeName="kubernetes.io/secret/de9d0ea2-cd91-46df-b5fb-f322312c3390-cert" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550381    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="519bdf1e-bdde-4785-a71c-74c0d80ea527" volumeName="kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550393    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5593ba-0e81-4d12-afbc-73019f33d7dd" volumeName="kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550404    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b19899c7-dc0f-439b-aa44-5948528cd810" volumeName="kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550417    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b19899c7-dc0f-439b-aa44-5948528cd810" volumeName="kubernetes.io/secret/b19899c7-dc0f-439b-aa44-5948528cd810-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550429    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9047a53-882d-4708-945a-663dcdc7dc9a" volumeName="kubernetes.io/configmap/f9047a53-882d-4708-945a-663dcdc7dc9a-konnectivity-ca" seLinuxMountContext=""
Nov 02 13:57:15.550438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550440    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3ecb7ef-ae15-4411-b6f2-5c515d9e3284" volumeName="kubernetes.io/configmap/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-iptables-alerter-script" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550452    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33196c6d-9ade-4612-99bd-4e757677ec89" volumeName="kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550464    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3" volumeName="kubernetes.io/projected/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-kube-api-access-xdlxx" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550482    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e9884e1-a236-4925-9875-53f6b1e64ba0" volumeName="kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-daemon-config" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550495    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e9884e1-a236-4925-9875-53f6b1e64ba0" volumeName="kubernetes.io/projected/6e9884e1-a236-4925-9875-53f6b1e64ba0-kube-api-access-m9hdn" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550681    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b19899c7-dc0f-439b-aa44-5948528cd810" volumeName="kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-env-overrides" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550694    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3ecb7ef-ae15-4411-b6f2-5c515d9e3284" volumeName="kubernetes.io/projected/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-kube-api-access-264tr" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550706    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5593ba-0e81-4d12-afbc-73019f33d7dd" volumeName="kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550721    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f928dc3-3c2e-4baa-b29b-de0a84eed156" volumeName="kubernetes.io/projected/7f928dc3-3c2e-4baa-b29b-de0a84eed156-kube-api-access-vfzfl" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550735    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbf753aa-a9de-4930-8d1e-546dcbf74281" volumeName="kubernetes.io/secret/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-key" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550747    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9047a53-882d-4708-945a-663dcdc7dc9a" volumeName="kubernetes.io/secret/f9047a53-882d-4708-945a-663dcdc7dc9a-agent-certs" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550760    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b" volumeName="kubernetes.io/projected/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-kube-api-access-brhbn" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550773    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33196c6d-9ade-4612-99bd-4e757677ec89" volumeName="kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550786    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b19899c7-dc0f-439b-aa44-5948528cd810" volumeName="kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-config" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550800    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b19899c7-dc0f-439b-aa44-5948528cd810" volumeName="kubernetes.io/projected/b19899c7-dc0f-439b-aa44-5948528cd810-kube-api-access-hj42k" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550813    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de9d0ea2-cd91-46df-b5fb-f322312c3390" volumeName="kubernetes.io/projected/de9d0ea2-cd91-46df-b5fb-f322312c3390-kube-api-access-5jfpn" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550826    4492 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b" volumeName="kubernetes.io/configmap/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-config-volume" seLinuxMountContext=""
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550839    4492 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550847    4492 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550710    4492 factory.go:153] Registering CRI-O factory
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550921    4492 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.550985    4492 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.551006    4492 factory.go:103] Registering Raw factory
Nov 02 13:57:15.551251 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.551019    4492 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:57:15.552019 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.551415    4492 manager.go:319] Starting recovery of all containers
Nov 02 13:57:15.571113 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.570997    4492 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:57:15.572833 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.572802    4492 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:57:15.572833 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.572838    4492 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:57:15.573021 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.572861    4492 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:57:15.573021 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:15.572898    4492 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 13:57:15.588935 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.588903    4492 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:57:15.588935 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.588940    4492 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 13:57:15.599030 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.599005    4492 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:57:15.599030 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.599016    4492 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:57:15.599272 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.599174    4492 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:57:15.603029 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.602997    4492 generic.go:356] "Generic (PLEG): container finished" podID="b15570f366559ffb67f868c74c61ae83" containerID="f2eb5f4483a84ca34d675fc4a62d37c4dcab5b7eb9130f27d8df84a325f7dfa8" exitCode=0
Nov 02 13:57:15.612674 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.612644    4492 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="e80eeca17b5f8936d2d558ea077b01293cc85b611cd1fe917e047923d90a5b21" exitCode=0
Nov 02 13:57:15.612674 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.612669    4492 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="72479cb6ee28775ff2c61dee512ee2b5a40f5183ed0a6c9a8329d0418e8b47de" exitCode=0
Nov 02 13:57:15.612674 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.612677    4492 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="325eeeda5e88835d02e1c340afbe346a59b248b9791442367d72de67c84fc2b1" exitCode=0
Nov 02 13:57:15.612674 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.612682    4492 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="5f9306f60a54349174445c9344744b0edd285036a4d9b4c13d018468cdba6116" exitCode=0
Nov 02 13:57:15.612932 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.612688    4492 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="fc3f5e02fe25e504c5b4e8ceb1a2758146c8e413badc47b9830ad6cda928d6e8" exitCode=0
Nov 02 13:57:15.612932 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.612693    4492 generic.go:356] "Generic (PLEG): container finished" podID="6d5593ba-0e81-4d12-afbc-73019f33d7dd" containerID="1f872a7480fb8c999e4e3c5caeb1fcc18b58049e15f7b95927b179b98856044d" exitCode=0
Nov 02 13:57:15.625448 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.625427    4492 manager.go:324] Recovery completed
Nov 02 13:57:15.641364 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641326    4492 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:57:15.641364 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641342    4492 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:57:15.641531 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641378    4492 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:57:15.641531 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641505    4492 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 13:57:15.641531 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641517    4492 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 13:57:15.641609 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641533    4492 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 13:57:15.641609 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641539    4492 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 13:57:15.641609 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641545    4492 policy_none.go:50] "Start"
Nov 02 13:57:15.641609 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641561    4492 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:57:15.641609 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641572    4492 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:57:15.641781 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641702    4492 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 13:57:15.641781 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641709    4492 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 13:57:15.641781 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.641715    4492 policy_none.go:44] "Start"
Nov 02 13:57:15.645617 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.645565    4492 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:57:15.648164 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.648149    4492 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:57:15.648844 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:15.648175    4492 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:57:15.648905 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.648850    4492 server.go:87] "Starting device plugin registration server"
Nov 02 13:57:15.649175 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649163    4492 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:57:15.649205 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649179    4492 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:57:15.649279 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649258    4492 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:57:15.649422 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649385    4492 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:57:15.649515 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649434    4492 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:57:15.649515 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649442    4492 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:57:15.649606 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.649500    4492 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:57:15.649420268Z","UUID":"39720dd0-8598-4781-a6c6-afdb0ef823e3","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:57:15.650765 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:15.650748    4492 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:57:15.650851 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.650782    4492 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:57:15.650851 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.650808    4492 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:57:15.673100 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.673028    4492 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"]
Nov 02 13:57:15.674183 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.674140    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" event={"ID":"b15570f366559ffb67f868c74c61ae83","Type":"ContainerStarted","Data":"cb23286c13f306a08be571f1b497b87fb5db18ea3f9a53ba71cc80c5315c3dda"}
Nov 02 13:57:15.674296 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.674201    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" event={"ID":"b15570f366559ffb67f868c74c61ae83","Type":"ContainerDied","Data":"f2eb5f4483a84ca34d675fc4a62d37c4dcab5b7eb9130f27d8df84a325f7dfa8"}
Nov 02 13:57:15.674296 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.674226    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal" event={"ID":"b15570f366559ffb67f868c74c61ae83","Type":"ContainerStarted","Data":"1c2be530e14a037dfc1fe0519160b0070c3b22e673f230fdb710c9d0357397bb"}
Nov 02 13:57:15.674296 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.674255    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal" event={"ID":"aff7ff2bf177bc815214d878993baa38","Type":"ContainerStarted","Data":"05718734b1cb9787b07ab7b872da7ea7e54bdcfa4b390054b422a607b9853b72"}
Nov 02 13:57:15.674296 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.674274    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal" event={"ID":"aff7ff2bf177bc815214d878993baa38","Type":"ContainerStarted","Data":"86267f65d17c300b016a56e0b76b9f891fe2145bef41226ada412ddc44eed168"}
Nov 02 13:57:15.674617 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.674598    4492 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.675082 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.675062    4492 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.694598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.694572    4492 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:57:15.694736 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:15.694639    4492 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.695709 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.695694    4492 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:57:15.695762 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:15.695732    4492 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.749446 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.749411    4492 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:57:15.751133 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.751114    4492 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:57:15.751221 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.751150    4492 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:57:15.751221 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.751163    4492 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-115.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:57:15.751221 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.751201    4492 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.781856 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.781822    4492 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.782014 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.781906    4492 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.789775 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.789740    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aff7ff2bf177bc815214d878993baa38-config\") pod \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" (UID: \"aff7ff2bf177bc815214d878993baa38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.789940 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.789788    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.789940 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.789825    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.890445 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.890404    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.890445 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.890450    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.890693 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.890468    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aff7ff2bf177bc815214d878993baa38-config\") pod \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" (UID: \"aff7ff2bf177bc815214d878993baa38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.890693 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.890544    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.890693 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.890569    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aff7ff2bf177bc815214d878993baa38-config\") pod \"kube-apiserver-proxy-ip-10-0-129-115.ec2.internal\" (UID: \"aff7ff2bf177bc815214d878993baa38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:15.890693 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:15.890551    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b15570f366559ffb67f868c74c61ae83-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal\" (UID: \"b15570f366559ffb67f868c74c61ae83\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal"
Nov 02 13:57:16.520958 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.520916    4492 apiserver.go:52] "Watching apiserver"
Nov 02 13:57:16.556456 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.556415    4492 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:57:16.556884 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.556857    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-wn8xj","openshift-service-ca/service-ca-5fffc97bdf-s228h","openshift-multus/multus-88ssr","openshift-cluster-node-tuning-operator/tuned-9xdrt","openshift-network-diagnostics/network-check-target-8cskr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-115.ec2.internal","openshift-network-operator/iptables-alerter-m622v","openshift-dns/node-resolver-t2wc7","openshift-multus/multus-additional-cni-plugins-wqm7n","openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj","kube-system/global-pull-secret-syncer-csrlb","kube-system/konnectivity-agent-2sbs7","openshift-insights/insights-runtime-extractor-6mkt2","openshift-insights/periodic-gathering-c4htm-7cd77","kube-system/kube-apiserver-proxy-ip-10-0-129-115.ec2.internal","openshift-multus/network-metrics-daemon-zjfbv","openshift-ingress-canary/ingress-canary-8r8vr","openshift-ovn-kubernetes/ovnkube-node-g557m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg","openshift-dns/dns-default-bpnnm"]
Nov 02 13:57:16.557320 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.557302    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:16.561559 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.561537    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.569432 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.569403    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.572842 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.572817    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.573048 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.573024    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:57:16.573140 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.573037    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm2nq\""
Nov 02 13:57:16.573140 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.573081    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.573939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.573922    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:57:16.574144 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574129    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5vzv\""
Nov 02 13:57:16.574210 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574162    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.574210 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574169    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.574758 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574731    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2zrsm\""
Nov 02 13:57:16.574758 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574736    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.574892 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574782    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.574892 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.574811    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.575036 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575009    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:57:16.575219 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575205    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlrs4\""
Nov 02 13:57:16.575282 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575250    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75928\""
Nov 02 13:57:16.575371 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575338    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gjwh\""
Nov 02 13:57:16.575542 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575449    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.575542 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575477    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.575542 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575479    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.575754 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.575739    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:57:16.576113 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.576101    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8ms52\""
Nov 02 13:57:16.576205 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.576182    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.576262 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.576218    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:57:16.576262 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.576244    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.576341 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.576263    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.579739 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.579716    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:57:16.580967 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.580951    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:57:16.581039 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.580970    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kftg9\""
Nov 02 13:57:16.581039 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.580976    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:57:16.581039 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.580998    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sczcr\""
Nov 02 13:57:16.581211 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.581194    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:57:16.582791 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.582774    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.583458 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.583442    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:57:16.584109 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.584092    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:57:16.593406 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.593376    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:57:16.593921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.593876    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cd6m4\""
Nov 02 13:57:16.593921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.593897    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lfskl\""
Nov 02 13:57:16.593921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.593898    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjjmq\""
Nov 02 13:57:16.593921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.593929    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:57:16.594279 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.593959    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jk68j\""
Nov 02 13:57:16.594279 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594041    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9cgsx\""
Nov 02 13:57:16.594279 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594049    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9rw8\""
Nov 02 13:57:16.594279 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594078    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mgxc\""
Nov 02 13:57:16.594279 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594082    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:57:16.594649 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594519    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 13:57:16.594649 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594545    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:57:16.594761 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594664    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4wpm\""
Nov 02 13:57:16.594761 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.594679    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5mnrf\""
Nov 02 13:57:16.636370 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.636334    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:16.650301 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.650278    4492 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:57:16.681047 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681018    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 13:57:16.681223 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681134    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.681223 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681161    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:57:16.681223 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681167    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.681223 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681174    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:57:16.681223 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681175    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.681223 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681201    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681245    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681276    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681316    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681348    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681372    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681389    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681390    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681457    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:57:16.681565 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681493    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.681948 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.681590    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:57:16.682090 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.682077    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 13:57:16.696277 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696246    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-node-log\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.696440 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696286    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-env-overrides\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.696440 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696312    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.696508 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696439    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.696538 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696512    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-tmp-dir\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.696538 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696525    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-env-overrides\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.696614 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696588    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-tmp-dir\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.696832 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696803    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.696977 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.696959    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-kubelet\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.697017 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697002    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-etc-kubernetes\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.697055 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697041    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.697097 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697084    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.697136 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697125    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jfpn\" (UniqueName: \"kubernetes.io/projected/de9d0ea2-cd91-46df-b5fb-f322312c3390-kube-api-access-5jfpn\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:16.697184 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697169    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a62f730c-7d26-4b82-a8cc-4607533c8a5e-crio-socket\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.697284 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697270    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a62f730c-7d26-4b82-a8cc-4607533c8a5e-data-volume\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.697394 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697298    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-netns\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.697394 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697372    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a62f730c-7d26-4b82-a8cc-4607533c8a5e-data-volume\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.697394 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697348    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-multus\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.697490 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697428    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqdc\" (UniqueName: \"kubernetes.io/projected/2d33d686-5e0b-4c04-b06c-3270a104521d-kube-api-access-8xqdc\") pod \"migrator-66df5d496b-q4phj\" (UID: \"2d33d686-5e0b-4c04-b06c-3270a104521d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:16.697490 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697478    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-slash\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.697587 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697574    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.697645 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697630    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-conf\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.697680 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697661    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-tmp\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.697717 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697689    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brhbn\" (UniqueName: \"kubernetes.io/projected/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-kube-api-access-brhbn\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.697747 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697716    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9047a53-882d-4708-945a-663dcdc7dc9a-konnectivity-ca\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:57:16.697747 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697736    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-tmp\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.697829 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697765    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-host\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.697880 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697860    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.697921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697892    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9047a53-882d-4708-945a-663dcdc7dc9a-konnectivity-ca\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:57:16.697921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697900    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd07b61c-b867-47ee-ae89-d324cbcaf032-serviceca\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.697994 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697932    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-iptables-alerter-script\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.697994 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.697969    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-hostroot\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.698068 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698049    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-multus-certs\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.698125 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698104    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hdn\" (UniqueName: \"kubernetes.io/projected/6e9884e1-a236-4925-9875-53f6b1e64ba0-kube-api-access-m9hdn\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.698181 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698160    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.698237 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698219    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-iptables-alerter-script\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.698308 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698293    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.698383 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698331    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.698447 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698387    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqdd\" (UniqueName: \"kubernetes.io/projected/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-api-access-2gqdd\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.698447 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698413    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-systemd\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.698556 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698539    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-run\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.698599 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698573    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9047a53-882d-4708-945a-663dcdc7dc9a-agent-certs\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:57:16.698651 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698632    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd07b61c-b867-47ee-ae89-d324cbcaf032-serviceca\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.698688 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698644    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-var-lib-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.698741 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698725    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-script-lib\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.698802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698788    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.698861 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698848    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.698861 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698844    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9047a53-882d-4708-945a-663dcdc7dc9a-agent-certs\") pod \"konnectivity-agent-2sbs7\" (UID: \"f9047a53-882d-4708-945a-663dcdc7dc9a\") " pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:57:16.698954 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698890    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.698954 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698945    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-script-lib\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.699049 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.698993    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-kubernetes\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.699049 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699016    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.699049 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699030    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cnibin\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.699160 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699052    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.699160 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699099    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-socket-dir-parent\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.699160 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699132    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-k8s-cni-cncf-io\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.699160 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699156    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.699333 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699199    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:16.699333 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699228    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzfl\" (UniqueName: \"kubernetes.io/projected/7f928dc3-3c2e-4baa-b29b-de0a84eed156-kube-api-access-vfzfl\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:16.699333 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699251    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-system-cni-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.699333 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699267    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpdb\" (UniqueName: \"kubernetes.io/projected/dd07b61c-b867-47ee-ae89-d324cbcaf032-kube-api-access-5jpdb\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.699333 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699294    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.699333 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699329    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-cabundle\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:16.699598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699412    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-cni-binary-copy\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.699598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699449    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.699598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699497    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-ovn\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.699598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699526    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5mh\" (UniqueName: \"kubernetes.io/projected/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kube-api-access-5n5mh\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.699598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699549    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de9d0ea2-cd91-46df-b5fb-f322312c3390-cert\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:16.699598 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699576    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699604    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699624    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-cabundle\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699628    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-system-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699667    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-sys\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699711    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-cni-binary-copy\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699722    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccx5\" (UniqueName: \"kubernetes.io/projected/cec63cc5-8483-4e4f-aca4-2160dab7a670-kube-api-access-8ccx5\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699776    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de9d0ea2-cd91-46df-b5fb-f322312c3390-cert\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:16.699871 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699790    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n479m\" (UniqueName: \"kubernetes.io/projected/6d5593ba-0e81-4d12-afbc-73019f33d7dd-kube-api-access-n479m\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.700087 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699926    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264tr\" (UniqueName: \"kubernetes.io/projected/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-kube-api-access-264tr\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.700087 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.699969    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.700087 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700006    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.700087 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700034    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-tmp-dir\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.700256 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700101    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-config-volume\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.700256 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700177    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.700256 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700201    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-tmp-dir\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.700385 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700332    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-kubelet-config\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.700385 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700379    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-config-volume\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.700488 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700396    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-dbus\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.700488 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700444    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.700587 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700509    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-netns\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.700587 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700549    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-config\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.700668 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700587    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b19899c7-dc0f-439b-aa44-5948528cd810-ovn-node-metrics-cert\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.700668 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700613    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-key\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:16.700668 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700641    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.700802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700696    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-log-socket\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.700802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700760    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b19899c7-dc0f-439b-aa44-5948528cd810-ovnkube-config\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.700802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700783    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/519bdf1e-bdde-4785-a71c-74c0d80ea527-original-pull-secret\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.700802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700797    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-modprobe-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.700970 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700813    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysconfig\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.700970 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700830    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.700970 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.700848    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-hosts-file\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.701100 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701078    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbf753aa-a9de-4930-8d1e-546dcbf74281-signing-key\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:16.701219 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701201    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b19899c7-dc0f-439b-aa44-5948528cd810-ovn-node-metrics-cert\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701219 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701210    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlxx\" (UniqueName: \"kubernetes.io/projected/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-kube-api-access-xdlxx\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.701299 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701254    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-daemon-config\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.701299 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701286    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-systemd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701418 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701314    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-bin\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701418 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701339    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.701418 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701343    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-netd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701562 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701418    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj42k\" (UniqueName: \"kubernetes.io/projected/b19899c7-dc0f-439b-aa44-5948528cd810-kube-api-access-hj42k\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701562 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701454    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-host-slash\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.701562 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701491    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv62\" (UniqueName: \"kubernetes.io/projected/dbf753aa-a9de-4930-8d1e-546dcbf74281-kube-api-access-4jv62\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:16.701562 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701535    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-daemon-config\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.701724 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701654    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701724 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701712    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd07b61c-b867-47ee-ae89-d324cbcaf032-host\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.701842 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701786    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-cnibin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.701889 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701838    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-kubelet\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.701889 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701870    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-device-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.701978 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701899    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-lib-modules\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.701978 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701931    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-var-lib-kubelet\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.701978 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701963    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-tuned\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.702102 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.701991    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-systemd-units\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.702102 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.702018    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.702102 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.702045    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-bin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.702102 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.702075    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-conf-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.702267 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.702103    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-etc-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.702346 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.702329    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-tuned\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.721950 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.721913    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jfpn\" (UniqueName: \"kubernetes.io/projected/de9d0ea2-cd91-46df-b5fb-f322312c3390-kube-api-access-5jfpn\") pod \"ingress-canary-8r8vr\" (UID: \"de9d0ea2-cd91-46df-b5fb-f322312c3390\") " pod="openshift-ingress-canary/ingress-canary-8r8vr"
Nov 02 13:57:16.730197 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.730174    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccx5\" (UniqueName: \"kubernetes.io/projected/cec63cc5-8483-4e4f-aca4-2160dab7a670-kube-api-access-8ccx5\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.731051 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.731026    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brhbn\" (UniqueName: \"kubernetes.io/projected/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-kube-api-access-brhbn\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.738771 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.738747    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqdc\" (UniqueName: \"kubernetes.io/projected/2d33d686-5e0b-4c04-b06c-3270a104521d-kube-api-access-8xqdc\") pod \"migrator-66df5d496b-q4phj\" (UID: \"2d33d686-5e0b-4c04-b06c-3270a104521d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q4phj"
Nov 02 13:57:16.738963 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.738944    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hdn\" (UniqueName: \"kubernetes.io/projected/6e9884e1-a236-4925-9875-53f6b1e64ba0-kube-api-access-m9hdn\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.739821 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.739799    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpdb\" (UniqueName: \"kubernetes.io/projected/dd07b61c-b867-47ee-ae89-d324cbcaf032-kube-api-access-5jpdb\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.739915 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.739899    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264tr\" (UniqueName: \"kubernetes.io/projected/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-kube-api-access-264tr\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.739993 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.739976    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzfl\" (UniqueName: \"kubernetes.io/projected/7f928dc3-3c2e-4baa-b29b-de0a84eed156-kube-api-access-vfzfl\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:16.740784 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.740764    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ggh\" (UniqueName: \"kubernetes.io/projected/8e854bb3-3994-496f-bc04-3ace19a132a9-kube-api-access-t5ggh\") pod \"network-check-target-8cskr\" (UID: \"8e854bb3-3994-496f-bc04-3ace19a132a9\") " pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:16.742459 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.742441    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj42k\" (UniqueName: \"kubernetes.io/projected/b19899c7-dc0f-439b-aa44-5948528cd810-kube-api-access-hj42k\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.744129 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.744081    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqdd\" (UniqueName: \"kubernetes.io/projected/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-api-access-2gqdd\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.744641 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.744623    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlxx\" (UniqueName: \"kubernetes.io/projected/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-kube-api-access-xdlxx\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.745221 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.745204    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n479m\" (UniqueName: \"kubernetes.io/projected/6d5593ba-0e81-4d12-afbc-73019f33d7dd-kube-api-access-n479m\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.745731 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.745714    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5mh\" (UniqueName: \"kubernetes.io/projected/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kube-api-access-5n5mh\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.747055 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.747040    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv62\" (UniqueName: \"kubernetes.io/projected/dbf753aa-a9de-4930-8d1e-546dcbf74281-kube-api-access-4jv62\") pod \"service-ca-5fffc97bdf-s228h\" (UID: \"dbf753aa-a9de-4930-8d1e-546dcbf74281\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s228h"
Nov 02 13:57:16.747143 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.747128    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") pod \"periodic-gathering-c4htm-7cd77\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") " pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:57:16.803475 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803347    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:16.803475 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803440    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-system-cni-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.803475 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803467    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803491    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:16.803503    4492 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803543    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-system-cni-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803553    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-ovn\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803574    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:16.803593    4492 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:16.803607    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.303583808 +0000 UTC m=+1.847299399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : secret "metrics-daemon-secret" not found
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:16.803651    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.303635493 +0000 UTC m=+1.847351065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803512    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-ovn\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803686    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-system-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803703    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-sys\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.803728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803726    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-kubelet-config\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803746    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-dbus\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803761    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-netns\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803787    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-log-socket\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803802    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-modprobe-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803802    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-system-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803815    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-dbus\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803821    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysconfig\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803805    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/519bdf1e-bdde-4785-a71c-74c0d80ea527-kubelet-config\") pod \"global-pull-secret-syncer-csrlb\" (UID: \"519bdf1e-bdde-4785-a71c-74c0d80ea527\") " pod="kube-system/global-pull-secret-syncer-csrlb"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803842    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-sys\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803852    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-netns\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803877    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysconfig\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803898    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-hosts-file\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803912    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-log-socket\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803919    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-modprobe-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803921    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-systemd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803942    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-bin\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803946    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-run-systemd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803949    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3-hosts-file\") pod \"node-resolver-t2wc7\" (UID: \"3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3\") " pod="openshift-dns/node-resolver-t2wc7"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803959    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-netd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.803996    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-netd\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804008    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-cni-bin\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804022    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-host-slash\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804058    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804085    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd07b61c-b867-47ee-ae89-d324cbcaf032-host\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804107    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-cnibin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804119    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804125    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3ecb7ef-ae15-4411-b6f2-5c515d9e3284-host-slash\") pod \"iptables-alerter-m622v\" (UID: \"b3ecb7ef-ae15-4411-b6f2-5c515d9e3284\") " pod="openshift-network-operator/iptables-alerter-m622v"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804129    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-kubelet\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804153    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-cnibin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804154    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-device-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804126    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd07b61c-b867-47ee-ae89-d324cbcaf032-host\") pod \"node-ca-wn8xj\" (UID: \"dd07b61c-b867-47ee-ae89-d324cbcaf032\") " pod="openshift-image-registry/node-ca-wn8xj"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804162    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-kubelet\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804171    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-lib-modules\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804186    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-var-lib-kubelet\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804210    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-systemd-units\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804222    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-var-lib-kubelet\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804229    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804230    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-device-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804237    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-lib-modules\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804251    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-systemd-units\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.804228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804260    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-bin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804274    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804282    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-bin\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804284    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-conf-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804302    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-conf-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804308    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-etc-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804330    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-node-log\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804372    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804392    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-etc-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804399    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-kubelet\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804404    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-node-log\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804423    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804422    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-etc-kubernetes\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804452    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804459    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-etc-kubernetes\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804467    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804493    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-run-ovn-kubernetes\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804495    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a62f730c-7d26-4b82-a8cc-4607533c8a5e-crio-socket\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804502    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804497    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-kubelet\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804523    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-netns\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804528    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a62f730c-7d26-4b82-a8cc-4607533c8a5e-crio-socket\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804546    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-multus\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804572    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-slash\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804586    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-netns\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804588    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-var-lib-cni-multus\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804595    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804626    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-host-slash\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804630    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-conf\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804689    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-host\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804715    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804722    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-d\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804741    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-hostroot\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804771    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-host\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804788    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-hostroot\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804790    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804805    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-multus-certs\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804825    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804843    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-multus-certs\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804844    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-sysctl-conf\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804876    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-systemd\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804843    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-systemd\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:16.804904    4492 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804910    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-run\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804937    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-var-lib-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804979    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.804992    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b19899c7-dc0f-439b-aa44-5948528cd810-var-lib-openvswitch\") pod \"ovnkube-node-g557m\" (UID: \"b19899c7-dc0f-439b-aa44-5948528cd810\") " pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:16.805007    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.304993727 +0000 UTC m=+1.848709299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805027    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805035    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-run\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805038    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805044    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-kubernetes\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805059    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cnibin\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805076    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805091    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cec63cc5-8483-4e4f-aca4-2160dab7a670-etc-kubernetes\") pod \"tuned-9xdrt\" (UID: \"cec63cc5-8483-4e4f-aca4-2160dab7a670\") " pod="openshift-cluster-node-tuning-operator/tuned-9xdrt"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805112    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5593ba-0e81-4d12-afbc-73019f33d7dd-cnibin\") pod \"multus-additional-cni-plugins-wqm7n\" (UID: \"6d5593ba-0e81-4d12-afbc-73019f33d7dd\") " pod="openshift-multus/multus-additional-cni-plugins-wqm7n"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805131    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-cni-dir\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805149    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-socket-dir-parent\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805169    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-k8s-cni-cncf-io\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805195    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805212    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-multus-socket-dir-parent\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805239    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6e9884e1-a236-4925-9875-53f6b1e64ba0-host-run-k8s-cni-cncf-io\") pod \"multus-88ssr\" (UID: \"6e9884e1-a236-4925-9875-53f6b1e64ba0\") " pod="openshift-multus/multus-88ssr"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805260    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f25f0e9a-bcc1-4ce1-9361-b98df140e5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-c2fxg\" (UID: \"f25f0e9a-bcc1-4ce1-9361-b98df140e5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c2fxg"
Nov 02 13:57:16.805749 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.805552    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a62f730c-7d26-4b82-a8cc-4607533c8a5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:16.867041 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:16.867010    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:17.199014 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.198987    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2sbs7"
Nov 02 13:57:17.277914 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.277884    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:17.309063 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.309029    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:17.309218 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.309070    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:17.309218 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.309117    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:17.309218 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:17.309180    4492 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:57:17.309218 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:17.309200    4492 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:17.309333 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:17.309237    4492 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:17.309333 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:17.309256    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.309239476 +0000 UTC m=+2.852955052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : secret "metrics-daemon-secret" not found
Nov 02 13:57:17.309333 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:17.309277    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.309265447 +0000 UTC m=+2.852981043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:17.309333 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:17.309290    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.309284547 +0000 UTC m=+2.853000119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:17.556601 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.556533    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t2wc7_3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3/dns-node-resolver/0.log"
Nov 02 13:57:17.686778 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.686746    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:17.880465 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:17.880390    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:18.078245 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.078215    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:18.314961 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.314928    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:18.314961 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.314968    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.314987    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:18.315093    4492 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:18.315093    4492 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:18.315140    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls podName:2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b nodeName:}" failed. No retries permitted until 2026-11-02 13:57:20.31512774 +0000 UTC m=+4.858843312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls") pod "dns-default-bpnnm" (UID: "2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b") : secret "dns-default-metrics-tls" not found
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:18.315093    4492 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:18.315154    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:20.315146679 +0000 UTC m=+4.858862250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:18.315176 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:18.315175    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:20.315162895 +0000 UTC m=+4.858878466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : secret "metrics-daemon-secret" not found
Nov 02 13:57:18.463331 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.463294    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:18.585135 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.585056    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:18.728252 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.728216    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:18.756700 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:18.756674    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wn8xj_dd07b61c-b867-47ee-ae89-d324cbcaf032/node-ca/0.log"
Nov 02 13:57:19.262154 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:19.262123    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:19.356997 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:19.356956    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8r8vr_de9d0ea2-cd91-46df-b5fb-f322312c3390/serve-healthcheck-canary/0.log"
Nov 02 13:57:19.862112 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:19.862063    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:19.945218 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:19.945193    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:19.957887 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:19.957864    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-q4phj_2d33d686-5e0b-4c04-b06c-3270a104521d/migrator/0.log"
Nov 02 13:57:20.057824 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.057774    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:20.172623 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.172538    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:20.331695 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.331655    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:20.331695 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.331692    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:20.331904 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.331740    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:20.331904 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:20.331804    4492 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:57:20.331904 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:20.331823    4492 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:20.331904 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:20.331875    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:24.331858524 +0000 UTC m=+8.875574099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : secret "metrics-daemon-secret" not found
Nov 02 13:57:20.331904 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:20.331891    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:24.331884627 +0000 UTC m=+8.875600202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:20.332153 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.332118    4492 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:57:20.334282 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.334263    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b-metrics-tls\") pod \"dns-default-bpnnm\" (UID: \"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b\") " pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:20.461526 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.461499    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:20.462342 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.462323    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:20.660907 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.660873    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:20.757866 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.757779    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bpnnm"]
Nov 02 13:57:20.763260 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:20.763224    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1db4ad_c5c2_4a5d_966a_641a4a3e1b0b.slice/crio-993cb5c435a0b5b360745371cc9e69aab08e82b072a96e922445cc7705405abe WatchSource:0}: Error finding container 993cb5c435a0b5b360745371cc9e69aab08e82b072a96e922445cc7705405abe: Status 404 returned error can't find the container with id 993cb5c435a0b5b360745371cc9e69aab08e82b072a96e922445cc7705405abe
Nov 02 13:57:20.765476 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:20.765458    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:57:21.260992 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:21.260922    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:21.634242 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:21.634162    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bpnnm" event={"ID":"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b","Type":"ContainerStarted","Data":"993cb5c435a0b5b360745371cc9e69aab08e82b072a96e922445cc7705405abe"}
Nov 02 13:57:21.659688 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:21.659649    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:22.058331 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:22.058301    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:22.462235 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:22.462205    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:22.641653 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:22.641563    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bpnnm" event={"ID":"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b","Type":"ContainerStarted","Data":"48b0cc96d771e932ac8f8eddd160c890010f248ac552c19314b503d4139d54d4"}
Nov 02 13:57:22.641653 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:22.641638    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bpnnm" event={"ID":"2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b","Type":"ContainerStarted","Data":"d5c28bd0b8191619b2d1e185b5cc674d77dc86f89bdf4a8d6c44940c35b72856"}
Nov 02 13:57:22.642027 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:22.641706    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:22.874620 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:22.874585    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:23.261688 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:23.261655    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:23.661788 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:23.661710    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:23.774778 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:23.774734    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:24.063186 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.063150    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:24.261505 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.261478    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:24.361917 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.361828    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:24.361917 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.361878    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:24.362146 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:24.361981    4492 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:57:24.362146 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:24.361996    4492 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:24.362146 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:24.362043    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs podName:7f928dc3-3c2e-4baa-b29b-de0a84eed156 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:32.362026259 +0000 UTC m=+16.905741852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs") pod "network-metrics-daemon-zjfbv" (UID: "7f928dc3-3c2e-4baa-b29b-de0a84eed156") : secret "metrics-daemon-secret" not found
Nov 02 13:57:24.362146 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:24.362072    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls podName:a62f730c-7d26-4b82-a8cc-4607533c8a5e nodeName:}" failed. No retries permitted until 2026-11-02 13:57:32.362054307 +0000 UTC m=+16.905769880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mkt2" (UID: "a62f730c-7d26-4b82-a8cc-4607533c8a5e") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:57:24.404287 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.404261    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:24.404499 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.404469    4492 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:57:24.863926 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:24.863893    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:25.836795 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:25.836763    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:26.058741 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:26.058709    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:26.263270 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:26.263236    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:26.685438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:26.685404    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:26.796320 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:26.796284    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:26.864642 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:26.864607    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:26.999178 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:26.999100    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:27.147395 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:27.147343    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:27.342760 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:27.342654    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:27.494371 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:27.494317    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:27.664996 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:27.664895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:27.785585 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:27.785538    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:27.899442 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:27.899414    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:28.013229 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:28.013197    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:28.197315 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:28.197284    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:28.308484 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:28.308415    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:29.683154 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:29.683117    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:29.796212 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:29.796181    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:31.538016 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:31.537984    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:31.647023 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:31.646992    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:32.078849 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.078815    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:32.418741 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.418647    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:32.418741 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.418701    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:32.421184 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.421150    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f928dc3-3c2e-4baa-b29b-de0a84eed156-metrics-certs\") pod \"network-metrics-daemon-zjfbv\" (UID: \"7f928dc3-3c2e-4baa-b29b-de0a84eed156\") " pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:32.421310 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.421208    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a62f730c-7d26-4b82-a8cc-4607533c8a5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mkt2\" (UID: \"a62f730c-7d26-4b82-a8cc-4607533c8a5e\") " pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:32.458289 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.458258    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjfbv"
Nov 02 13:57:32.470494 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.470467    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mkt2"
Nov 02 13:57:32.646593 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.646563    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bpnnm"
Nov 02 13:57:32.759537 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.759505    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fcc8bf4f4-8djlj"]
Nov 02 13:57:32.762787 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.762766    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zjfbv"]
Nov 02 13:57:32.762911 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.762847    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.770657 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:32.770619    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f928dc3_3c2e_4baa_b29b_de0a84eed156.slice/crio-22f082dfc266c1d7e1cccd8288cde6676ed90e7ef341b634d93463f41202e929 WatchSource:0}: Error finding container 22f082dfc266c1d7e1cccd8288cde6676ed90e7ef341b634d93463f41202e929: Status 404 returned error can't find the container with id 22f082dfc266c1d7e1cccd8288cde6676ed90e7ef341b634d93463f41202e929
Nov 02 13:57:32.784544 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.784521    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:32.790561 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:32.790535    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62f730c_7d26_4b82_a8cc_4607533c8a5e.slice/crio-63f63677d800fbd626a247813e50f355999fb1b4b6548ca2a3cac726fecfb134 WatchSource:0}: Error finding container 63f63677d800fbd626a247813e50f355999fb1b4b6548ca2a3cac726fecfb134: Status 404 returned error can't find the container with id 63f63677d800fbd626a247813e50f355999fb1b4b6548ca2a3cac726fecfb134
Nov 02 13:57:32.800332 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.800310    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 13:57:32.800686 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.800665    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 13:57:32.800784 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.800670    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 13:57:32.800784 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.800722    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mfgzk\""
Nov 02 13:57:32.800891 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.800857    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 13:57:32.801161 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.801142    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mkt2"]
Nov 02 13:57:32.801526 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.801511    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fcc8bf4f4-8djlj"]
Nov 02 13:57:32.918074 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.918040    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:32.923097 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923077    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9251af26-a06c-4b19-a480-181565db89f9-trusted-ca\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923168 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923106    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-bound-sa-token\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923168 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923125    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbxg\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-kube-api-access-6tbxg\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923241 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923208    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-registry-tls\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923292 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923278    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9251af26-a06c-4b19-a480-181565db89f9-image-registry-private-configuration\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923327 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923301    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9251af26-a06c-4b19-a480-181565db89f9-ca-trust-extracted\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923390 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923371    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9251af26-a06c-4b19-a480-181565db89f9-registry-certificates\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:32.923438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:32.923397    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9251af26-a06c-4b19-a480-181565db89f9-installation-pull-secrets\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024108 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024026    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9251af26-a06c-4b19-a480-181565db89f9-image-registry-private-configuration\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024108 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024062    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9251af26-a06c-4b19-a480-181565db89f9-ca-trust-extracted\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024108 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024078    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9251af26-a06c-4b19-a480-181565db89f9-registry-certificates\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024108 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024093    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9251af26-a06c-4b19-a480-181565db89f9-installation-pull-secrets\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024477 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024129    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9251af26-a06c-4b19-a480-181565db89f9-trusted-ca\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024477 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024257    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-bound-sa-token\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024477 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024306    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbxg\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-kube-api-access-6tbxg\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024477 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024381    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-registry-tls\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.024688 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.024523    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9251af26-a06c-4b19-a480-181565db89f9-ca-trust-extracted\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.025194 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.025167    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9251af26-a06c-4b19-a480-181565db89f9-trusted-ca\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.025306 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.025249    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9251af26-a06c-4b19-a480-181565db89f9-registry-certificates\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.026874 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.026848    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9251af26-a06c-4b19-a480-181565db89f9-installation-pull-secrets\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.026967 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.026884    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9251af26-a06c-4b19-a480-181565db89f9-image-registry-private-configuration\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.027186 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.027171    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-registry-tls\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.045856 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.045819    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-bound-sa-token\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.045856 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.045842    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbxg\" (UniqueName: \"kubernetes.io/projected/9251af26-a06c-4b19-a480-181565db89f9-kube-api-access-6tbxg\") pod \"image-registry-fcc8bf4f4-8djlj\" (UID: \"9251af26-a06c-4b19-a480-181565db89f9\") " pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.050423 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.050402    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:33.075511 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.075484    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.382964 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.382886    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fcc8bf4f4-8djlj"]
Nov 02 13:57:33.674955 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.674916    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj" event={"ID":"9251af26-a06c-4b19-a480-181565db89f9","Type":"ContainerStarted","Data":"69b166781a0eae9fc34a1104d6082779e0b3711a115bd91d0391ad35920f53cb"}
Nov 02 13:57:33.675449 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.674969    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj" event={"ID":"9251af26-a06c-4b19-a480-181565db89f9","Type":"ContainerStarted","Data":"0b51477e372689e05ae471534931094ad261d3fdd0a72a52dedd02539b15d398"}
Nov 02 13:57:33.675449 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.674995    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:33.676767 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.676742    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mkt2" event={"ID":"a62f730c-7d26-4b82-a8cc-4607533c8a5e","Type":"ContainerStarted","Data":"4820cdc1517ec6f7857c8a3b13879fc9dd55822070a14753576ac79f3954c90f"}
Nov 02 13:57:33.676890 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.676790    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mkt2" event={"ID":"a62f730c-7d26-4b82-a8cc-4607533c8a5e","Type":"ContainerStarted","Data":"863837dce070aed35321f0ca111191caa933f8047ee3956803e92bd8aaff1cea"}
Nov 02 13:57:33.676890 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.676815    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mkt2" event={"ID":"a62f730c-7d26-4b82-a8cc-4607533c8a5e","Type":"ContainerStarted","Data":"63f63677d800fbd626a247813e50f355999fb1b4b6548ca2a3cac726fecfb134"}
Nov 02 13:57:33.677847 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.677817    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjfbv" event={"ID":"7f928dc3-3c2e-4baa-b29b-de0a84eed156","Type":"ContainerStarted","Data":"22f082dfc266c1d7e1cccd8288cde6676ed90e7ef341b634d93463f41202e929"}
Nov 02 13:57:33.717666 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:33.717607    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj" podStartSLOduration=1.717591939 podStartE2EDuration="1.717591939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:33.717106546 +0000 UTC m=+18.260822141" watchObservedRunningTime="2026-11-02 13:57:33.717591939 +0000 UTC m=+18.261307530"
Nov 02 13:57:34.683773 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:34.683728    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjfbv" event={"ID":"7f928dc3-3c2e-4baa-b29b-de0a84eed156","Type":"ContainerStarted","Data":"650e0bb4a423ddf346ace749e5f2a76ea6ef3daf8c2b4549a15c7b763cd030aa"}
Nov 02 13:57:34.684237 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:34.683798    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjfbv" event={"ID":"7f928dc3-3c2e-4baa-b29b-de0a84eed156","Type":"ContainerStarted","Data":"d46c09322242059e4a063f371b6c8414eb46c37108f7b640796e718d2f717041"}
Nov 02 13:57:35.507785 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:35.507744    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:35.650518 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:35.650471    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:35.687968 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:35.687931    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mkt2" event={"ID":"a62f730c-7d26-4b82-a8cc-4607533c8a5e","Type":"ContainerStarted","Data":"99d00488e0b4eb7870c29fd29f429bd3557576c3109b9e44b814e32e5c4627ce"}
Nov 02 13:57:35.808842 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:35.808762    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:38.768777 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:38.768745    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:39.944316 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:39.944282    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:40.189647 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.189610    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:40.277546 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.277472    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lj7gg"]
Nov 02 13:57:40.281681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.281658    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.298701 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.298678    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:57:40.308551 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.308529    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:57:40.308551 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.308548    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 13:57:40.308941 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.308925    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 13:57:40.331022 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.330997    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:40.356076 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.356050    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z8rpr\""
Nov 02 13:57:40.356206 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.356173    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 13:57:40.356380 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.356348    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 13:57:40.371689 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371658    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-root\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.371818 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371694    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-sys\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.371818 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371732    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-wtmp\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.371917 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371822    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-textfile\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.371917 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371864    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-tls\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.371917 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371898    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-accelerators-collector-config\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.372048 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371964    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.372048 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.371998    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84a4265a-c047-424e-b69c-c5ca1ff31395-metrics-client-ca\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.372148 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.372085    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdlm\" (UniqueName: \"kubernetes.io/projected/84a4265a-c047-424e-b69c-c5ca1ff31395-kube-api-access-bgdlm\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.451373 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.451313    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:40.472413 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472387    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-wtmp\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472507 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472417    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-textfile\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472507 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472434    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-tls\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472507 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472453    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-accelerators-collector-config\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472621 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472561    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-wtmp\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472621 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472568    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472696 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472626    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84a4265a-c047-424e-b69c-c5ca1ff31395-metrics-client-ca\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472696 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472666    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdlm\" (UniqueName: \"kubernetes.io/projected/84a4265a-c047-424e-b69c-c5ca1ff31395-kube-api-access-bgdlm\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472763 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472695    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-root\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472763 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472720    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-sys\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472838 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472784    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-sys\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472838 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472820    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-textfile\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.472924 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.472884    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/84a4265a-c047-424e-b69c-c5ca1ff31395-root\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.473039 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.473018    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-accelerators-collector-config\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.473144 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.473126    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84a4265a-c047-424e-b69c-c5ca1ff31395-metrics-client-ca\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.474966 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.474945    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-tls\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.475097 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.475078    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84a4265a-c047-424e-b69c-c5ca1ff31395-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.499061 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.499032    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdlm\" (UniqueName: \"kubernetes.io/projected/84a4265a-c047-424e-b69c-c5ca1ff31395-kube-api-access-bgdlm\") pod \"node-exporter-lj7gg\" (UID: \"84a4265a-c047-424e-b69c-c5ca1ff31395\") " pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.597287 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.597222    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lj7gg"
Nov 02 13:57:40.608111 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:40.608085    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a4265a_c047_424e_b69c_c5ca1ff31395.slice/crio-98829557a1d03664a28e8687761d8bd2f23f48001ca5bbbc972620552e97f2ae WatchSource:0}: Error finding container 98829557a1d03664a28e8687761d8bd2f23f48001ca5bbbc972620552e97f2ae: Status 404 returned error can't find the container with id 98829557a1d03664a28e8687761d8bd2f23f48001ca5bbbc972620552e97f2ae
Nov 02 13:57:40.644979 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.644955    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:40.704618 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.704584    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lj7gg" event={"ID":"84a4265a-c047-424e-b69c-c5ca1ff31395","Type":"ContainerStarted","Data":"98829557a1d03664a28e8687761d8bd2f23f48001ca5bbbc972620552e97f2ae"}
Nov 02 13:57:40.821146 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:40.821113    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:41.413721 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:41.413680    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:41.528102 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:41.528050    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:41.709155 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:41.709124    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lj7gg" event={"ID":"84a4265a-c047-424e-b69c-c5ca1ff31395","Type":"ContainerStarted","Data":"3c77fd872cdcda46f8f1fae082d5a78526fde553fb7ebccf49dade0eaed71031"}
Nov 02 13:57:41.769945 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:41.769918    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:42.048512 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.048444    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g557m"
Nov 02 13:57:42.511296 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.511263    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"]
Nov 02 13:57:42.514831 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.514810    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.530921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.530897    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 13:57:42.531085 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.530952    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 13:57:42.531085 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.531020    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 13:57:42.531085 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.531077    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 13:57:42.531206 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.531089    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7bt9pjtk95jub\""
Nov 02 13:57:42.531206 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.531096    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p76cp\""
Nov 02 13:57:42.531610 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.531593    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 13:57:42.552493 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.552467    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"]
Nov 02 13:57:42.690506 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690461    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-grpc-tls\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690518    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwf7\" (UniqueName: \"kubernetes.io/projected/943f3fe6-e595-4e27-a5e9-79c855043ae2-kube-api-access-gpwf7\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690612    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-tls\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690640    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690661    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690809 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690720    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943f3fe6-e595-4e27-a5e9-79c855043ae2-metrics-client-ca\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690809 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690764    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943f3fe6-e595-4e27-a5e9-79c855043ae2-tmp\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690809 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690792    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.690939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.690819    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.715292 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.715245    4492 generic.go:356] "Generic (PLEG): container finished" podID="84a4265a-c047-424e-b69c-c5ca1ff31395" containerID="3c77fd872cdcda46f8f1fae082d5a78526fde553fb7ebccf49dade0eaed71031" exitCode=0
Nov 02 13:57:42.715459 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.715382    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lj7gg" event={"ID":"84a4265a-c047-424e-b69c-c5ca1ff31395","Type":"ContainerDied","Data":"3c77fd872cdcda46f8f1fae082d5a78526fde553fb7ebccf49dade0eaed71031"}
Nov 02 13:57:42.791317 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791288    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943f3fe6-e595-4e27-a5e9-79c855043ae2-metrics-client-ca\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791452 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791327    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943f3fe6-e595-4e27-a5e9-79c855043ae2-tmp\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791452 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791371    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791452 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791400    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791452 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791422    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-grpc-tls\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791452 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791445    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwf7\" (UniqueName: \"kubernetes.io/projected/943f3fe6-e595-4e27-a5e9-79c855043ae2-kube-api-access-gpwf7\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791466    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-tls\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791484    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791681 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791513    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.791802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.791727    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/943f3fe6-e595-4e27-a5e9-79c855043ae2-tmp\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.792184 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.792118    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/943f3fe6-e595-4e27-a5e9-79c855043ae2-metrics-client-ca\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.794221 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.794195    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.794321 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.794238    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-tls\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.794566 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.794544    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.794640 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.794619    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.794721 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.794705    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.794984 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.794968    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/943f3fe6-e595-4e27-a5e9-79c855043ae2-secret-grpc-tls\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.818019 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.817994    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwf7\" (UniqueName: \"kubernetes.io/projected/943f3fe6-e595-4e27-a5e9-79c855043ae2-kube-api-access-gpwf7\") pod \"thanos-querier-77cc7cbd6f-78cnn\" (UID: \"943f3fe6-e595-4e27-a5e9-79c855043ae2\") " pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.837201 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.837181    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:42.878295 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.878270    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:42.997594 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:42.997561    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:43.164864 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.164767    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"]
Nov 02 13:57:43.181787 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:43.181757    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943f3fe6_e595_4e27_a5e9_79c855043ae2.slice/crio-3bf5c66226df7cdfe3cb359fc8eb08d0c3d3318935ae6eb24c89723bd2827f16 WatchSource:0}: Error finding container 3bf5c66226df7cdfe3cb359fc8eb08d0c3d3318935ae6eb24c89723bd2827f16: Status 404 returned error can't find the container with id 3bf5c66226df7cdfe3cb359fc8eb08d0c3d3318935ae6eb24c89723bd2827f16
Nov 02 13:57:43.509213 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.509185    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:43.636224 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.636191    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:43.721464 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.721427    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lj7gg" event={"ID":"84a4265a-c047-424e-b69c-c5ca1ff31395","Type":"ContainerStarted","Data":"3afa07931f58c9905c77c6b83548f9980f03af0d95293cadd332ae0d3e104563"}
Nov 02 13:57:43.721660 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.721490    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lj7gg" event={"ID":"84a4265a-c047-424e-b69c-c5ca1ff31395","Type":"ContainerStarted","Data":"cf6e831cd0ae7c4eda1bdff5b071dae25e0efdf9acc216170484f1073b37c9be"}
Nov 02 13:57:43.722714 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.722682    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"3bf5c66226df7cdfe3cb359fc8eb08d0c3d3318935ae6eb24c89723bd2827f16"}
Nov 02 13:57:43.751424 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.751402    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:43.872432 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.872338    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:43.999391 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:43.999337    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:44.471919 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.471883    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:44.809439 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.809320    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lj7gg" podStartSLOduration=3.773065803 podStartE2EDuration="4.8093039s" totalImagesPullingTime="962.691143ms" totalInitContainerRuntime="73.546954ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:43.76400143 +0000 UTC m=+28.307717024" watchObservedRunningTime="2026-11-02 13:57:44.8093039 +0000 UTC m=+29.353019484"
Nov 02 13:57:44.809894 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.809848    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"]
Nov 02 13:57:44.827802 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.827777    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:44.838899 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.838876    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"]
Nov 02 13:57:44.846735 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.846683    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4m9mx\""
Nov 02 13:57:44.852293 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.852268    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-doabg81df27hp\""
Nov 02 13:57:44.852293 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.852287    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 13:57:44.852293 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.852273    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 13:57:44.852506 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.852314    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 13:57:44.852506 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:44.852406    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 13:57:45.010269 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010224    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wwp\" (UniqueName: \"kubernetes.io/projected/67121397-06f1-4c97-bc11-8d26c7a9a5c1-kube-api-access-q8wwp\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.010269 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010275    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-secret-metrics-server-tls\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.010516 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010321    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/67121397-06f1-4c97-bc11-8d26c7a9a5c1-metrics-server-audit-profiles\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.010516 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010443    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-client-ca-bundle\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.010516 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010490    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67121397-06f1-4c97-bc11-8d26c7a9a5c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.010659 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010518    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/67121397-06f1-4c97-bc11-8d26c7a9a5c1-audit-log\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.010659 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.010557    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-secret-metrics-server-client-certs\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.101567 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.101472    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:45.111582 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111551    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wwp\" (UniqueName: \"kubernetes.io/projected/67121397-06f1-4c97-bc11-8d26c7a9a5c1-kube-api-access-q8wwp\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.111735 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111597    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-secret-metrics-server-tls\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.111735 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111638    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/67121397-06f1-4c97-bc11-8d26c7a9a5c1-metrics-server-audit-profiles\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.111735 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111670    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-client-ca-bundle\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.111735 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111710    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67121397-06f1-4c97-bc11-8d26c7a9a5c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.111948 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111735    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/67121397-06f1-4c97-bc11-8d26c7a9a5c1-audit-log\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.111948 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.111770    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-secret-metrics-server-client-certs\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.112971 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.112947    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/67121397-06f1-4c97-bc11-8d26c7a9a5c1-audit-log\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.113118 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.112979    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67121397-06f1-4c97-bc11-8d26c7a9a5c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.113299 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.113274    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/67121397-06f1-4c97-bc11-8d26c7a9a5c1-metrics-server-audit-profiles\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.114804 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.114776    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-secret-metrics-server-client-certs\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.114804 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.114802    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-client-ca-bundle\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.114939 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.114819    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/67121397-06f1-4c97-bc11-8d26c7a9a5c1-secret-metrics-server-tls\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.135326 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.135301    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wwp\" (UniqueName: \"kubernetes.io/projected/67121397-06f1-4c97-bc11-8d26c7a9a5c1-kube-api-access-q8wwp\") pod \"metrics-server-6cff5b96cc-wgq5m\" (UID: \"67121397-06f1-4c97-bc11-8d26c7a9a5c1\") " pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.155300 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.155055    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:57:45.181608 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.181571    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"]
Nov 02 13:57:45.198785 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.198751    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.221314 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.221177    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 13:57:45.221314 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.221287    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h2gc9\""
Nov 02 13:57:45.222621 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.222593    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"]
Nov 02 13:57:45.252235 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.252203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:45.313400 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.313367    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35cae0b0-182b-4ea1-bf0b-9ac313533d9b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9r7zl\" (UID: \"35cae0b0-182b-4ea1-bf0b-9ac313533d9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.313541 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.313432    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvnm\" (UniqueName: \"kubernetes.io/projected/35cae0b0-182b-4ea1-bf0b-9ac313533d9b-kube-api-access-7wvnm\") pod \"monitoring-plugin-77f4f76594-9r7zl\" (UID: \"35cae0b0-182b-4ea1-bf0b-9ac313533d9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.402438 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.402370    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:45.414533 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.414511    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvnm\" (UniqueName: \"kubernetes.io/projected/35cae0b0-182b-4ea1-bf0b-9ac313533d9b-kube-api-access-7wvnm\") pod \"monitoring-plugin-77f4f76594-9r7zl\" (UID: \"35cae0b0-182b-4ea1-bf0b-9ac313533d9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.414616 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.414546    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35cae0b0-182b-4ea1-bf0b-9ac313533d9b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9r7zl\" (UID: \"35cae0b0-182b-4ea1-bf0b-9ac313533d9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.416931 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.416914    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35cae0b0-182b-4ea1-bf0b-9ac313533d9b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9r7zl\" (UID: \"35cae0b0-182b-4ea1-bf0b-9ac313533d9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.435829 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.435804    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvnm\" (UniqueName: \"kubernetes.io/projected/35cae0b0-182b-4ea1-bf0b-9ac313533d9b-kube-api-access-7wvnm\") pod \"monitoring-plugin-77f4f76594-9r7zl\" (UID: \"35cae0b0-182b-4ea1-bf0b-9ac313533d9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.498898 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.498862    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"]
Nov 02 13:57:45.537806 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.537770    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:45.674551 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:45.674526    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67121397_06f1_4c97_bc11_8d26c7a9a5c1.slice/crio-1bddfbc16acad35d4814f05356d6ed08842649c8098d5eb078cf796d64d06e5f WatchSource:0}: Error finding container 1bddfbc16acad35d4814f05356d6ed08842649c8098d5eb078cf796d64d06e5f: Status 404 returned error can't find the container with id 1bddfbc16acad35d4814f05356d6ed08842649c8098d5eb078cf796d64d06e5f
Nov 02 13:57:45.730444 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:45.730413    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m" event={"ID":"67121397-06f1-4c97-bc11-8d26c7a9a5c1","Type":"ContainerStarted","Data":"1bddfbc16acad35d4814f05356d6ed08842649c8098d5eb078cf796d64d06e5f"}
Nov 02 13:57:46.001503 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.001476    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"]
Nov 02 13:57:46.007067 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:57:46.007037    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cae0b0_182b_4ea1_bf0b_9ac313533d9b.slice/crio-cc7cbb466bbcad22ac1e242dd0dab0d22cd4d629146c327e670369b8ce1f067f WatchSource:0}: Error finding container cc7cbb466bbcad22ac1e242dd0dab0d22cd4d629146c327e670369b8ce1f067f: Status 404 returned error can't find the container with id cc7cbb466bbcad22ac1e242dd0dab0d22cd4d629146c327e670369b8ce1f067f
Nov 02 13:57:46.521759 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:57:46.521721    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 13:57:46.546460 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.546423    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:46.680702 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.680661    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:46.740678 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.740638    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"b59a157eec7fcca1bd55ae2513c6918ae656f5b939a46c08bc122581098501ec"}
Nov 02 13:57:46.740868 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.740731    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"a7f2052ddb1303c8c8c45ed752d62a919140c32a4ffc6a7fbe82b26e466ce2d9"}
Nov 02 13:57:46.740868 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.740772    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"1a0ae8e95ebbe066eb848c6f4699158924235b86bc7d1fcb952f73c30d6657e5"}
Nov 02 13:57:46.742674 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.742643    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl" event={"ID":"35cae0b0-182b-4ea1-bf0b-9ac313533d9b","Type":"ContainerStarted","Data":"cc7cbb466bbcad22ac1e242dd0dab0d22cd4d629146c327e670369b8ce1f067f"}
Nov 02 13:57:46.799777 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:46.799705    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8cskr"
Nov 02 13:57:47.632908 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:47.632871    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:47.751786 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:47.751723    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"ccda1b5d1a6c7a3af3e7c6fa84bfdfc3118c21d5396447309825cd443fe89b39"}
Nov 02 13:57:47.751945 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:47.751838    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"dda0ecf6b4341f50c351fa958e5d02311ba1495f6d6def401b04308fe95c7166"}
Nov 02 13:57:47.753912 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:47.753885    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m" event={"ID":"67121397-06f1-4c97-bc11-8d26c7a9a5c1","Type":"ContainerStarted","Data":"0130558e7b54f4de20ed79888ffd269292e361216d1332d3f82a73bea1920f00"}
Nov 02 13:57:47.808072 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:47.808012    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m" podStartSLOduration=2.073971605 podStartE2EDuration="3.807993464s" totalImagesPullingTime="1.734021859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:47.807154408 +0000 UTC m=+32.350870002" watchObservedRunningTime="2026-11-02 13:57:47.807993464 +0000 UTC m=+32.351709051"
Nov 02 13:57:47.933892 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:47.933853    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:48.760869 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:48.760833    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" event={"ID":"943f3fe6-e595-4e27-a5e9-79c855043ae2","Type":"ContainerStarted","Data":"f5038451480449f958ea7281a972850d2bf1f5b08912b0b3aec9e6c5ade4a8ba"}
Nov 02 13:57:48.761296 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:48.760974    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:48.762412 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:48.762383    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl" event={"ID":"35cae0b0-182b-4ea1-bf0b-9ac313533d9b","Type":"ContainerStarted","Data":"7441c7eaccaa2a98c8db9cd7ef79db3b45de97dd7e8dba97b9c40a82ec0f7acd"}
Nov 02 13:57:48.762703 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:48.762686    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:48.767264 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:48.767240    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl"
Nov 02 13:57:48.877954 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:48.877894    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn" podStartSLOduration=2.916161229 podStartE2EDuration="6.877874366s" totalImagesPullingTime="3.961713137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:48.832457852 +0000 UTC m=+33.376173446" watchObservedRunningTime="2026-11-02 13:57:48.877874366 +0000 UTC m=+33.421589962"
Nov 02 13:57:50.944230 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:50.944203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:51.069827 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:51.069798    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:53.089102 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:53.089034    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:53.207968 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:53.207935    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:54.688742 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:54.688712    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fcc8bf4f4-8djlj"
Nov 02 13:57:54.746938 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:54.746888    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9r7zl" podStartSLOduration=7.555022856 podStartE2EDuration="9.74687209s" totalImagesPullingTime="2.191849234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:48.877237705 +0000 UTC m=+33.420953320" watchObservedRunningTime="2026-11-02 13:57:54.74687209 +0000 UTC m=+39.290587683"
Nov 02 13:57:54.773451 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:54.773423    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77cc7cbd6f-78cnn"
Nov 02 13:57:54.931624 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:54.931588    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:55.047304 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:55.047212    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:55.160627 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:55.160592    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:55.434302 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:55.434269    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:55.546911 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:55.546873    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:58.091206 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:58.091128    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:58.212588 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:57:58.212556    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:01.535306 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:01.535271    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:01.649837 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:01.649804    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:02.699869 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:02.699836    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:02.860902 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:02.860875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:04.211673 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:04.211640    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:04.323201 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:04.323168    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:04.658843 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:04.658764    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:04.801278 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:04.801248    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:05.155920 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:05.155883    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:58:05.156081 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:05.155950    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:58:16.515563 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:58:16.515499    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 13:58:16.548143 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:16.548112    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:16.660728 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:16.660697    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:18.163240 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:18.163205    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:18.283005 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:18.282970    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:25.160922 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:25.160886    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:58:25.164656 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:25.164635    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cff5b96cc-wgq5m"
Nov 02 13:58:26.029087 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:26.029042    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:26.145337 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:26.145287    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:26.264602 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:26.264556    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:30.738428 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:30.738397    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-c4htm-7cd77"]
Nov 02 13:58:30.738920 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:30.738812    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-c4htm-7cd77" podUID="33196c6d-9ade-4612-99bd-4e757677ec89" containerName="insights-gathering" containerID="cri-o://fe6b3e95565d55f7b634a335d16a77aff6aecf4f0c9f0ea927d65944ddff3caf" gracePeriod=30
Nov 02 13:58:30.895925 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:30.895893    4492 generic.go:356] "Generic (PLEG): container finished" podID="33196c6d-9ade-4612-99bd-4e757677ec89" containerID="fe6b3e95565d55f7b634a335d16a77aff6aecf4f0c9f0ea927d65944ddff3caf" exitCode=2
Nov 02 13:58:30.896083 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:30.895964    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c4htm-7cd77" event={"ID":"33196c6d-9ade-4612-99bd-4e757677ec89","Type":"ContainerDied","Data":"fe6b3e95565d55f7b634a335d16a77aff6aecf4f0c9f0ea927d65944ddff3caf"}
Nov 02 13:58:30.982962 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:30.982940    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:58:31.039348 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039272    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") pod \"33196c6d-9ade-4612-99bd-4e757677ec89\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") "
Nov 02 13:58:31.039348 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039339    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") pod \"33196c6d-9ade-4612-99bd-4e757677ec89\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") "
Nov 02 13:58:31.039585 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039412    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") pod \"33196c6d-9ade-4612-99bd-4e757677ec89\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") "
Nov 02 13:58:31.039585 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039447    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") pod \"33196c6d-9ade-4612-99bd-4e757677ec89\" (UID: \"33196c6d-9ade-4612-99bd-4e757677ec89\") "
Nov 02 13:58:31.039693 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039658    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle" pod "33196c6d-9ade-4612-99bd-4e757677ec89" (UID: "33196c6d-9ade-4612-99bd-4e757677ec89"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:31.039752 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039728    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path" pod "33196c6d-9ade-4612-99bd-4e757677ec89" (UID: "33196c6d-9ade-4612-99bd-4e757677ec89"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:58:31.039752 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.039728    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle" pod "33196c6d-9ade-4612-99bd-4e757677ec89" (UID: "33196c6d-9ade-4612-99bd-4e757677ec89"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:31.041949 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.041919    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj" pod "33196c6d-9ade-4612-99bd-4e757677ec89" (UID: "33196c6d-9ade-4612-99bd-4e757677ec89"). InnerVolumeSpecName "kube-api-access-c42cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:31.140364 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.140329    4492 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-trusted-ca-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 13:58:31.140364 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.140368    4492 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/33196c6d-9ade-4612-99bd-4e757677ec89-archives-path\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 13:58:31.140543 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.140381    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c42cj\" (UniqueName: \"kubernetes.io/projected/33196c6d-9ade-4612-99bd-4e757677ec89-kube-api-access-c42cj\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 13:58:31.140543 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.140391    4492 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33196c6d-9ade-4612-99bd-4e757677ec89-service-ca-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 13:58:31.537113 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.537078    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:31.654835 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.654803    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:31.900531 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.900449    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c4htm-7cd77"
Nov 02 13:58:31.900531 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.900458    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c4htm-7cd77" event={"ID":"33196c6d-9ade-4612-99bd-4e757677ec89","Type":"ContainerDied","Data":"5e4dfec36a7973819c4aa6900db02f2fddb0f9b69f6162bb362c4e4ec5ccafb5"}
Nov 02 13:58:31.900531 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.900532    4492 scope.go:122] "RemoveContainer" containerID="fe6b3e95565d55f7b634a335d16a77aff6aecf4f0c9f0ea927d65944ddff3caf"
Nov 02 13:58:31.946368 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.946332    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-c4htm-7cd77"]
Nov 02 13:58:31.950600 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:31.950578    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-c4htm-7cd77"]
Nov 02 13:58:33.577880 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:33.577850    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33196c6d-9ade-4612-99bd-4e757677ec89" path="/var/lib/kubelet/pods/33196c6d-9ade-4612-99bd-4e757677ec89/volumes"
Nov 02 13:58:40.909002 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:40.908959    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:41.024041 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:41.024007    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:46.212166 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.212129    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-89c7555db-8rtqx"]
Nov 02 13:58:46.212608 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.212390    4492 cpu_manager.go:414] "containerMap: removing container" podUID="33196c6d-9ade-4612-99bd-4e757677ec89" containerName="insights-gathering"
Nov 02 13:58:46.212608 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.212401    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33196c6d-9ade-4612-99bd-4e757677ec89" containerName="insights-gathering"
Nov 02 13:58:46.212608 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.212484    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="33196c6d-9ade-4612-99bd-4e757677ec89" containerName="insights-gathering"
Nov 02 13:58:46.216982 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.216963    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.241195 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.241168    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 13:58:46.250245 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.250219    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 13:58:46.250391 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.250302    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2cv9s\""
Nov 02 13:58:46.250391 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.250342    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 13:58:46.252081 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.252061    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-89c7555db-8rtqx"]
Nov 02 13:58:46.305376 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.305332    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 13:58:46.315960 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.315934    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 13:58:46.343988 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.343964    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 13:58:46.352180 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352158    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352234 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352196    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-secret-telemeter-client\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352234 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352226    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-metrics-client-ca\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352335 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352316    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-serving-certs-ca-bundle\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352410 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352395    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-telemeter-client-tls\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352449 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352421    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352449 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352439    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-federate-client-tls\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.352513 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.352456    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbxt\" (UniqueName: \"kubernetes.io/projected/b9921d64-e034-4664-87d4-9e61f1234ac6-kube-api-access-kwbxt\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453619 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453587    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-secret-telemeter-client\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453738 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453634    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-metrics-client-ca\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453779 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453750    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-serving-certs-ca-bundle\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453809 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453792    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-telemeter-client-tls\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453845 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453826    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453877 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453852    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-federate-client-tls\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453915 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453880    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbxt\" (UniqueName: \"kubernetes.io/projected/b9921d64-e034-4664-87d4-9e61f1234ac6-kube-api-access-kwbxt\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.453976 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.453950    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.454517 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.454488    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-serving-certs-ca-bundle\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.454759 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.454739    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.454953 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.454931    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9921d64-e034-4664-87d4-9e61f1234ac6-metrics-client-ca\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.456253 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.456234    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-secret-telemeter-client\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.456604 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.456581    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-federate-client-tls\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.456784 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.456748    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.456894 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.456878    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b9921d64-e034-4664-87d4-9e61f1234ac6-telemeter-client-tls\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.478121 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.478061    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbxt\" (UniqueName: \"kubernetes.io/projected/b9921d64-e034-4664-87d4-9e61f1234ac6-kube-api-access-kwbxt\") pod \"telemeter-client-89c7555db-8rtqx\" (UID: \"b9921d64-e034-4664-87d4-9e61f1234ac6\") " pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.515021 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:58:46.514989    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 13:58:46.531921 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.531899    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx"
Nov 02 13:58:46.855687 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.855598    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-89c7555db-8rtqx"]
Nov 02 13:58:46.871587 ip-10-0-129-115 kubenswrapper[4492]: W1102 13:58:46.871563    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9921d64_e034_4664_87d4_9e61f1234ac6.slice/crio-a4e0708cc10bfddd6e8b97832f8692d61674c655b36cc9e06cb8490a1613348e WatchSource:0}: Error finding container a4e0708cc10bfddd6e8b97832f8692d61674c655b36cc9e06cb8490a1613348e: Status 404 returned error can't find the container with id a4e0708cc10bfddd6e8b97832f8692d61674c655b36cc9e06cb8490a1613348e
Nov 02 13:58:46.949606 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:46.949572    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx" event={"ID":"b9921d64-e034-4664-87d4-9e61f1234ac6","Type":"ContainerStarted","Data":"a4e0708cc10bfddd6e8b97832f8692d61674c655b36cc9e06cb8490a1613348e"}
Nov 02 13:58:48.958850 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:48.958817    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx" event={"ID":"b9921d64-e034-4664-87d4-9e61f1234ac6","Type":"ContainerStarted","Data":"0d0b00ce87c834f054789a21e930e947a370289896437d41a490cb83f6760128"}
Nov 02 13:58:49.964123 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:49.964091    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx" event={"ID":"b9921d64-e034-4664-87d4-9e61f1234ac6","Type":"ContainerStarted","Data":"82f6a0bd5491191464f84b14f29b61bfc121c9968e6843fca6f0d6062c9fdc5c"}
Nov 02 13:58:49.965447 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:49.964152    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx" event={"ID":"b9921d64-e034-4664-87d4-9e61f1234ac6","Type":"ContainerStarted","Data":"98f0c30c97e754d929ce7e651d9fab869dcf7ff2cc79025d2475b97c31c77c6d"}
Nov 02 13:58:50.036041 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:50.035989    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-89c7555db-8rtqx" podStartSLOduration=1.466207769 podStartE2EDuration="4.035976523s" totalImagesPullingTime="2.569768754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:50.024300011 +0000 UTC m=+94.568015615" watchObservedRunningTime="2026-11-02 13:58:50.035976523 +0000 UTC m=+94.579692117"
Nov 02 13:58:50.370781 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:50.370694    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:50.497157 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:50.497126    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:50.657228 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:50.657141    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:50.775545 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:58:50.775515    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:04.813647 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:04.813613    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:16.516253 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:59:16.516206    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 13:59:46.515714 ip-10-0-129-115 kubenswrapper[4492]: E1102 13:59:46.515671    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 13:59:55.310168 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:55.310133    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:56.897759 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:56.897727    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:57.007114 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:57.007076    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:58.062160 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:58.062123    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:58.172463 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:58.172431    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:58.646455 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:58.646416    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:58.758897 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:58.758867    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:58.960627 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:58.960599    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:59.074587 ip-10-0-129-115 kubenswrapper[4492]: I1102 13:59:59.074555    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:04.923842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:04.923812    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:09.508623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:09.508590    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:09.623241 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:09.623210    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:14.941378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:14.941321    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:16.516524 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:00:16.516482    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:00:16.557442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:16.557407    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:16.666943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:16.666916    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:30.960605 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:30.960568    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:31.073928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:31.073889    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:46.516719 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:00:46.516679    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:00:48.146783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:48.146750    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:51.925062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:51.925026    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:52.038669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:00:52.038632    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:05.050186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:01:05.050104    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:15.056712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:01:15.056678    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:16.515498 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:01:16.515464    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:01:44.888340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:01:44.888306    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:44.999714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:01:44.999684    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:46.515590 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:01:46.515554    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:02:05.177708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:05.177667    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:15.193504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:15.193470    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:16.515453 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:02:16.515413    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:02:29.420844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:29.420758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:30.456438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.456404    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:30.863597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.863500    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"]
Nov 02 14:02:30.866935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.866912    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:30.880516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.880494    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 14:02:30.891076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.891046    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 14:02:30.891254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.891114    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nmghn\""
Nov 02 14:02:30.902535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.902506    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"]
Nov 02 14:02:30.926835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.926797    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cls9\" (UniqueName: \"kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:30.927009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.926853    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:30.927009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:30.926969    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.023521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.023487    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:31.027587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.027569    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.027649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.027605    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.027649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.027639    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cls9\" (UniqueName: \"kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.027927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.027910    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.027963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.027950    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.055683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.055647    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cls9\" (UniqueName: \"kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.102816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.102784    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"]
Nov 02 14:02:31.106316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.106293    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.127845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.127780    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"]
Nov 02 14:02:31.128140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.128112    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.128276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.128162    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5qj\" (UniqueName: \"kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.128276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.128253    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.130195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.130172    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vkkcm\""
Nov 02 14:02:31.142643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.142596    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:31.181961 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.181923    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:02:31.229419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.229349    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.229548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.229418    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5qj\" (UniqueName: \"kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.229548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.229454    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.229798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.229772    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.230052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.230029    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.252967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.252935    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5qj\" (UniqueName: \"kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.422405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.422294    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:02:31.478224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.478157    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"]
Nov 02 14:02:31.487781 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:31.487747    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6494f5_2f58_436e_b7fe_bb4a36438502.slice/crio-b836c060a17e2f1a2693fd5742d744985cdf4600db157a6f7642fb30e5ef7bc1 WatchSource:0}: Error finding container b836c060a17e2f1a2693fd5742d744985cdf4600db157a6f7642fb30e5ef7bc1: Status 404 returned error can't find the container with id b836c060a17e2f1a2693fd5742d744985cdf4600db157a6f7642fb30e5ef7bc1
Nov 02 14:02:31.489828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.489798    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:02:31.692022 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.691990    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk" event={"ID":"ef6494f5-2f58-436e-b7fe-bb4a36438502","Type":"ContainerStarted","Data":"b836c060a17e2f1a2693fd5742d744985cdf4600db157a6f7642fb30e5ef7bc1"}
Nov 02 14:02:31.723840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:31.723805    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"]
Nov 02 14:02:31.728879 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:31.728852    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1638ae_03ea_4a0f_98fc_efa4d7305faa.slice/crio-4f7a71404f847c3c7fc2749f2421e96684fb621d58d590c043a25e43a8faadd3 WatchSource:0}: Error finding container 4f7a71404f847c3c7fc2749f2421e96684fb621d58d590c043a25e43a8faadd3: Status 404 returned error can't find the container with id 4f7a71404f847c3c7fc2749f2421e96684fb621d58d590c043a25e43a8faadd3
Nov 02 14:02:32.449206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:32.449171    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:32.698817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:32.698737    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t" event={"ID":"bc1638ae-03ea-4a0f-98fc-efa4d7305faa","Type":"ContainerStarted","Data":"4f7a71404f847c3c7fc2749f2421e96684fb621d58d590c043a25e43a8faadd3"}
Nov 02 14:02:33.777142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.777107    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"]
Nov 02 14:02:33.781046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.781022    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.806121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.806088    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-bgzfh\""
Nov 02 14:02:33.807772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.807708    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"]
Nov 02 14:02:33.853557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.853517    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.853758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.853590    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.853758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.853662    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspmz\" (UniqueName: \"kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.954340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.954303    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.954544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.954420    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspmz\" (UniqueName: \"kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.954544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.954459    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.954752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.954730    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.954803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.954780    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:33.981120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:33.981088    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspmz\" (UniqueName: \"kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:34.099659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:34.099574    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:02:34.405482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:34.405130    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"]
Nov 02 14:02:34.411510 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:34.411477    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6731c93_cd7d_45ef_ac36_7b10270bbfd2.slice/crio-71fc1299c99b1d02a9b1295f5e4526afe7e0af4841f773a8ac0d44a9ea20cf5c WatchSource:0}: Error finding container 71fc1299c99b1d02a9b1295f5e4526afe7e0af4841f773a8ac0d44a9ea20cf5c: Status 404 returned error can't find the container with id 71fc1299c99b1d02a9b1295f5e4526afe7e0af4841f773a8ac0d44a9ea20cf5c
Nov 02 14:02:34.709317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:34.709280    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx" event={"ID":"b6731c93-cd7d-45ef-ac36-7b10270bbfd2","Type":"ContainerStarted","Data":"71fc1299c99b1d02a9b1295f5e4526afe7e0af4841f773a8ac0d44a9ea20cf5c"}
Nov 02 14:02:34.860770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:34.860735    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:34.979602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:34.979518    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:35.147987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.147956    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"]
Nov 02 14:02:35.161150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.161118    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.189519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.189488    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:02:35.189519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.189488    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 14:02:35.189519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.189524    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4tfr8\""
Nov 02 14:02:35.189786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.189581    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:02:35.191422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.191398    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"]
Nov 02 14:02:35.237992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.237875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:35.266989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266595    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.266989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266642    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.266989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266671    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.266989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266701    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.266989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266733    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.266989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266771    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.267698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.266821    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.267698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.267538    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7w6r\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.267698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.267609    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.309151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.309118    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"]
Nov 02 14:02:35.313146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.312784    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.337268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.337229    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xg4t9\""
Nov 02 14:02:35.348819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.348747    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"]
Nov 02 14:02:35.370488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.370411    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a412529-a227-45f2-8124-9caad6a192a9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.372185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.370764    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.372185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.370825    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a412529-a227-45f2-8124-9caad6a192a9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.372185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372139    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.370871    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372498    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7w6r\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372530    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372572    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372621    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372655    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a412529-a227-45f2-8124-9caad6a192a9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372680    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sv4w\" (UniqueName: \"kubernetes.io/projected/0a412529-a227-45f2-8124-9caad6a192a9-kube-api-access-6sv4w\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372718    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372770    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372794    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372820    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372869    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.372998    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.373031    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.373071    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.373389    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.373640    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.373988    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.374037    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.378384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.377318    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.379745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.379682    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.402349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.402323    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"]
Nov 02 14:02:35.407384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.407328    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.408265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.408239    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.410744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.410720    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7w6r\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r\") pod \"maas-default-gateway-openshift-default-55fbd65575-clxsz\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.436456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.436426    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-bqpz4\""
Nov 02 14:02:35.441203 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.441172    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"]
Nov 02 14:02:35.447597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.447553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:35.473816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473679    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.473816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473732    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.473816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473764    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zs5\" (UniqueName: \"kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.473816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473793    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.473816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473823    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a412529-a227-45f2-8124-9caad6a192a9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473847    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sv4w\" (UniqueName: \"kubernetes.io/projected/0a412529-a227-45f2-8124-9caad6a192a9-kube-api-access-6sv4w\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473893    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473950    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.473979    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474019    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a412529-a227-45f2-8124-9caad6a192a9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474050    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a412529-a227-45f2-8124-9caad6a192a9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474093    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.474458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474149    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474230    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474404    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.474736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.474648    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.475038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.475011    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a412529-a227-45f2-8124-9caad6a192a9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.477220 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.477197    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a412529-a227-45f2-8124-9caad6a192a9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.478226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.478207    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:35.486941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.486916    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a412529-a227-45f2-8124-9caad6a192a9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.509127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.509058    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sv4w\" (UniqueName: \"kubernetes.io/projected/0a412529-a227-45f2-8124-9caad6a192a9-kube-api-access-6sv4w\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.510251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.510223    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a412529-a227-45f2-8124-9caad6a192a9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr\" (UID: \"0a412529-a227-45f2-8124-9caad6a192a9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.571182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.569712    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"]
Nov 02 14:02:35.576099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.575711    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.576099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.575776    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.576099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.575810    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zs5\" (UniqueName: \"kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.576548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.576310    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.576548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.576512    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.581628 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.581603    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.602207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.602153    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"]
Nov 02 14:02:35.603025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.603000    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8xg9k\""
Nov 02 14:02:35.611078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.611022    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zs5\" (UniqueName: \"kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.643504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.643471    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:35.644340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.644292    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:35.677248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.677216    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.677398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.677272    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.677398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.677316    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nb8r\" (UniqueName: \"kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.733458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.733414    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"]
Nov 02 14:02:35.737953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.737900    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.738296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.738272    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:02:35.751720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.751688    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-nwsb2\""
Nov 02 14:02:35.770509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.770476    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"]
Nov 02 14:02:35.778652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.778619    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.778674    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nb8r\" (UniqueName: \"kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.778722    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvrc\" (UniqueName: \"kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.778782    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.778824    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.778855    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.779267    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.780413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.779541    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.818238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.818165    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nb8r\" (UniqueName: \"kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.858628 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.858595    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"]
Nov 02 14:02:35.860903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.860876    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"]
Nov 02 14:02:35.864779 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:35.864697    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737bb7e1_871d_43d6_80b8_7a22ab0c1ffa.slice/crio-a96095d6c6efd651ecf64fd0b35c9d3a85a95aee38869ddcff5b117c26e09060 WatchSource:0}: Error finding container a96095d6c6efd651ecf64fd0b35c9d3a85a95aee38869ddcff5b117c26e09060: Status 404 returned error can't find the container with id a96095d6c6efd651ecf64fd0b35c9d3a85a95aee38869ddcff5b117c26e09060
Nov 02 14:02:35.865065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.865047    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:35.879632 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.879606    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7rr47\""
Nov 02 14:02:35.879986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.879966    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvrc\" (UniqueName: \"kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.880047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.880035    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.880091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.880079    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.880524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.880504    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.881253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.880890    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.889408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.889089    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"]
Nov 02 14:02:35.906596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.906565    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:02:35.931203 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.931171    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvrc\" (UniqueName: \"kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:35.980682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.980642    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvs6\" (UniqueName: \"kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:35.980798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.980699    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:35.980798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:35.980766    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.005769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.002676    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"]
Nov 02 14:02:36.066204 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.066165    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:02:36.071977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.071945    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"]
Nov 02 14:02:36.078927 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:36.078884    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c32f87e_206f_4ea4_aa83_d16e7f55fce6.slice/crio-53437d1ce014a4ff34620b825f5e0d92da256b762aaea9d084acae25fd6619e9 WatchSource:0}: Error finding container 53437d1ce014a4ff34620b825f5e0d92da256b762aaea9d084acae25fd6619e9: Status 404 returned error can't find the container with id 53437d1ce014a4ff34620b825f5e0d92da256b762aaea9d084acae25fd6619e9
Nov 02 14:02:36.082556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.082536    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.082649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.082628    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvs6\" (UniqueName: \"kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.082721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.082658    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.083059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.082992    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.083235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.083216    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.110758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.110717    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvs6\" (UniqueName: \"kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.186737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.186702    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:02:36.234025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.233025    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"]
Nov 02 14:02:36.388654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.388594    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"]
Nov 02 14:02:36.394150 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:36.394117    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9debe8bf_90e1_493f_aeb8_7c9984bcbcaf.slice/crio-0b56d9828f5c17390c19ab41ad2e9c6ecfa715c9054a73a87c21c977dba0b872 WatchSource:0}: Error finding container 0b56d9828f5c17390c19ab41ad2e9c6ecfa715c9054a73a87c21c977dba0b872: Status 404 returned error can't find the container with id 0b56d9828f5c17390c19ab41ad2e9c6ecfa715c9054a73a87c21c977dba0b872
Nov 02 14:02:36.515138 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.515040    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"]
Nov 02 14:02:36.520156 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:36.520122    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b666de9_372e_4659_902c_e8da861c6258.slice/crio-6b716d9e078095e63745aaf02e0ae5676d2971a7206ed0a970279f90862a44ae WatchSource:0}: Error finding container 6b716d9e078095e63745aaf02e0ae5676d2971a7206ed0a970279f90862a44ae: Status 404 returned error can't find the container with id 6b716d9e078095e63745aaf02e0ae5676d2971a7206ed0a970279f90862a44ae
Nov 02 14:02:36.727155 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.727113    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg" event={"ID":"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf","Type":"ContainerStarted","Data":"0b56d9828f5c17390c19ab41ad2e9c6ecfa715c9054a73a87c21c977dba0b872"}
Nov 02 14:02:36.729177 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.729145    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr" event={"ID":"0a412529-a227-45f2-8124-9caad6a192a9","Type":"ContainerStarted","Data":"cd539fbc9983014222fa42a45b5bf4204dbac92e5786459057796c7eaf517431"}
Nov 02 14:02:36.731038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.730988    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb" event={"ID":"96a75551-eb75-4769-9eb5-f841ff7cf738","Type":"ContainerStarted","Data":"aae9bf55d267696ea85a5f7fd3abc95d4d4a90c88b22b68fc294db700dc4c915"}
Nov 02 14:02:36.734050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.734028    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" event={"ID":"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa","Type":"ContainerStarted","Data":"a96095d6c6efd651ecf64fd0b35c9d3a85a95aee38869ddcff5b117c26e09060"}
Nov 02 14:02:36.736215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.736188    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn" event={"ID":"0b666de9-372e-4659-902c-e8da861c6258","Type":"ContainerStarted","Data":"6b716d9e078095e63745aaf02e0ae5676d2971a7206ed0a970279f90862a44ae"}
Nov 02 14:02:36.743743 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:36.743716    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" event={"ID":"9c32f87e-206f-4ea4-aa83-d16e7f55fce6","Type":"ContainerStarted","Data":"53437d1ce014a4ff34620b825f5e0d92da256b762aaea9d084acae25fd6619e9"}
Nov 02 14:02:37.036497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.034434    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"]
Nov 02 14:02:37.040241 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.040207    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.061273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.061245    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"]
Nov 02 14:02:37.092067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.091986    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.092067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.092039    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.092480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.092118    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzbm\" (UniqueName: \"kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.193601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.193562    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.193807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.193614    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzbm\" (UniqueName: \"kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.193807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.193773    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.194233 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.194200    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.194483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.194463    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.233906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.223282    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzbm\" (UniqueName: \"kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.374318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.374035    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:02:37.412627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.410417    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"]
Nov 02 14:02:37.426666 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.426636    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"]
Nov 02 14:02:37.427037 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.426898    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.508897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.508577    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.508897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.508644    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmh5\" (UniqueName: \"kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.508897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.508682    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.610070 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.610016    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.610260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.610092    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmh5\" (UniqueName: \"kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.610260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.610129    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.610687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.610659    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.610687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.610675    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.650265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.650176    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmh5\" (UniqueName: \"kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.728236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.728105    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"]
Nov 02 14:02:37.754386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.754337    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:02:37.780126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:37.780045    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb" event={"ID":"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44","Type":"ContainerStarted","Data":"7ce36883e7086fc3be54ce9720a03a46d194eab8bcc28f01006710dcc210c50f"}
Nov 02 14:02:38.102570 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:38.102524    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"]
Nov 02 14:02:38.112652 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:02:38.112601    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03286bc1_501b_4c3b_b464_90ea17aeb465.slice/crio-3ceaa7619bfca977326ae4890b0f29bc19a2f6a3cdb0a751295a1f468ca1f851 WatchSource:0}: Error finding container 3ceaa7619bfca977326ae4890b0f29bc19a2f6a3cdb0a751295a1f468ca1f851: Status 404 returned error can't find the container with id 3ceaa7619bfca977326ae4890b0f29bc19a2f6a3cdb0a751295a1f468ca1f851
Nov 02 14:02:38.400635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:38.400597    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:38.821630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:38.821525    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx" event={"ID":"03286bc1-501b-4c3b-b464-90ea17aeb465","Type":"ContainerStarted","Data":"3ceaa7619bfca977326ae4890b0f29bc19a2f6a3cdb0a751295a1f468ca1f851"}
Nov 02 14:02:45.542416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.542244    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:02:45.542416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.542334    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:02:45.542416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.542400    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:02:45.647030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.646972    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:02:45.647136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.647067    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:02:45.647136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.647109    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:02:45.936386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.935613    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr" event={"ID":"0a412529-a227-45f2-8124-9caad6a192a9","Type":"ContainerStarted","Data":"09c10beb0e7edf60245347a2a34fdc230c78c7dae5bd91a026b24b65af5694f0"}
Nov 02 14:02:45.944229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.942897    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" event={"ID":"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa","Type":"ContainerStarted","Data":"3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"}
Nov 02 14:02:45.999722 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:45.999658    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr" podStartSLOduration=1.468266554 podStartE2EDuration="10.999639629s" totalImagesPullingTime="9.531373075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:45.989641961 +0000 UTC m=+330.533357556" watchObservedRunningTime="2026-11-02 14:02:45.999639629 +0000 UTC m=+330.543355225"
Nov 02 14:02:46.034076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.034008    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podStartSLOduration=1.260831144 podStartE2EDuration="11.033987794s" totalImagesPullingTime="9.77315665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:46.032735442 +0000 UTC m=+330.576451244" watchObservedRunningTime="2026-11-02 14:02:46.033987794 +0000 UTC m=+330.577703390"
Nov 02 14:02:46.479529 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.479440    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:46.485505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.485207    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:46.560557 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:02:46.560489    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:02:46.645424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.645067    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:46.651369 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.651327    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:46.961107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.961067    4492 generic.go:356] "Generic (PLEG): container finished" podID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerID="684fef6dca8bb87282b23576f87bb4ecc7b49e2bf3474902c51ab2e308ad0903" exitCode=0
Nov 02 14:02:46.962002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.961971    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk" event={"ID":"ef6494f5-2f58-436e-b7fe-bb4a36438502","Type":"ContainerDied","Data":"684fef6dca8bb87282b23576f87bb4ecc7b49e2bf3474902c51ab2e308ad0903"}
Nov 02 14:02:46.966276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.966252    4492 generic.go:356] "Generic (PLEG): container finished" podID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerID="8db41e41639dbc9e07dea7dd90d8950183fbf5c6fa5c4964febc9d5766cc5357" exitCode=0
Nov 02 14:02:46.968616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.968593    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t" event={"ID":"bc1638ae-03ea-4a0f-98fc-efa4d7305faa","Type":"ContainerDied","Data":"8db41e41639dbc9e07dea7dd90d8950183fbf5c6fa5c4964febc9d5766cc5357"}
Nov 02 14:02:46.969800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.969780    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:46.969883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.969826    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:46.970983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.970965    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:02:46.971134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:46.971120    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr"
Nov 02 14:02:47.979375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:47.976566    4492 generic.go:356] "Generic (PLEG): container finished" podID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerID="ad48eb9430abc13829d196cbaf7be3c534a2682bacf5220ed7a4cedf38e270e0" exitCode=0
Nov 02 14:02:47.979375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:47.977804    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t" event={"ID":"bc1638ae-03ea-4a0f-98fc-efa4d7305faa","Type":"ContainerDied","Data":"ad48eb9430abc13829d196cbaf7be3c534a2682bacf5220ed7a4cedf38e270e0"}
Nov 02 14:02:48.986309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:48.986133    4492 generic.go:356] "Generic (PLEG): container finished" podID="0b666de9-372e-4659-902c-e8da861c6258" containerID="0f0b15344d90cb6d0218e7e939ee16759aa1b581296d97749ef5909226bfe9ff" exitCode=0
Nov 02 14:02:48.986309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:48.986244    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn" event={"ID":"0b666de9-372e-4659-902c-e8da861c6258","Type":"ContainerDied","Data":"0f0b15344d90cb6d0218e7e939ee16759aa1b581296d97749ef5909226bfe9ff"}
Nov 02 14:02:48.988587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:48.988560    4492 generic.go:356] "Generic (PLEG): container finished" podID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerID="9eeceaf162222a5a3b0440a0c9085b34e6e3e873cf525da30c6ad16eb17540cc" exitCode=0
Nov 02 14:02:48.988692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:48.988631    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" event={"ID":"9c32f87e-206f-4ea4-aa83-d16e7f55fce6","Type":"ContainerDied","Data":"9eeceaf162222a5a3b0440a0c9085b34e6e3e873cf525da30c6ad16eb17540cc"}
Nov 02 14:02:48.992739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:48.992705    4492 generic.go:356] "Generic (PLEG): container finished" podID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerID="d342045156a925af48f85b266122047faf4c66db20f2e8abb2e18001d791761a" exitCode=0
Nov 02 14:02:48.992841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:48.992815    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx" event={"ID":"03286bc1-501b-4c3b-b464-90ea17aeb465","Type":"ContainerDied","Data":"d342045156a925af48f85b266122047faf4c66db20f2e8abb2e18001d791761a"}
Nov 02 14:02:49.002228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:49.002188    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb" event={"ID":"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44","Type":"ContainerStarted","Data":"0ca2232ecd9d06ab7432f0cdb51dbf53306c00c207cc68a9d1ce782137a59c2f"}
Nov 02 14:02:49.005043 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:49.005014    4492 generic.go:356] "Generic (PLEG): container finished" podID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerID="61d1fb70899e80bd15c158b3a5f667f00a508b4844ef7fdd40920bcf02d8340d" exitCode=0
Nov 02 14:02:49.005166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:49.005100    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg" event={"ID":"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf","Type":"ContainerDied","Data":"61d1fb70899e80bd15c158b3a5f667f00a508b4844ef7fdd40920bcf02d8340d"}
Nov 02 14:02:49.010768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:49.007116    4492 generic.go:356] "Generic (PLEG): container finished" podID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerID="ad421b6380d0ea302f2971038c9ff5c8ed7992755175e230035407c6a18f1e56" exitCode=0
Nov 02 14:02:49.010768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:49.007322    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb" event={"ID":"96a75551-eb75-4769-9eb5-f841ff7cf738","Type":"ContainerDied","Data":"ad421b6380d0ea302f2971038c9ff5c8ed7992755175e230035407c6a18f1e56"}
Nov 02 14:02:50.026548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.026435    4492 generic.go:356] "Generic (PLEG): container finished" podID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerID="0ca2232ecd9d06ab7432f0cdb51dbf53306c00c207cc68a9d1ce782137a59c2f" exitCode=0
Nov 02 14:02:50.026548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.026484    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb" event={"ID":"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44","Type":"ContainerDied","Data":"0ca2232ecd9d06ab7432f0cdb51dbf53306c00c207cc68a9d1ce782137a59c2f"}
Nov 02 14:02:50.041650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.040309    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg" event={"ID":"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf","Type":"ContainerStarted","Data":"3072bd3e449a0d63ab21ee930e2895b445ae250b8e9795c6bd17cd3ebda6704b"}
Nov 02 14:02:50.043745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.043655    4492 generic.go:356] "Generic (PLEG): container finished" podID="0b666de9-372e-4659-902c-e8da861c6258" containerID="3fc2af5110e978348071e3220cb21be05d63d628130ef0bb3d75d94d02b6e7a6" exitCode=0
Nov 02 14:02:50.043820 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.043754    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn" event={"ID":"0b666de9-372e-4659-902c-e8da861c6258","Type":"ContainerDied","Data":"3fc2af5110e978348071e3220cb21be05d63d628130ef0bb3d75d94d02b6e7a6"}
Nov 02 14:02:50.049617 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.049587    4492 generic.go:356] "Generic (PLEG): container finished" podID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerID="90710f80f35695c3c23ddb263811a9d58d5a6112c911b03caad99153c258fc6c" exitCode=0
Nov 02 14:02:50.049826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:50.049631    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx" event={"ID":"03286bc1-501b-4c3b-b464-90ea17aeb465","Type":"ContainerDied","Data":"90710f80f35695c3c23ddb263811a9d58d5a6112c911b03caad99153c258fc6c"}
Nov 02 14:02:51.058007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.057951    4492 generic.go:356] "Generic (PLEG): container finished" podID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerID="ef19ef0c2679c46c4dfaaa8819a6f33598ab8d7023b4db65b760b1a2e3d2282c" exitCode=0
Nov 02 14:02:51.058528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.058058    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk" event={"ID":"ef6494f5-2f58-436e-b7fe-bb4a36438502","Type":"ContainerDied","Data":"ef19ef0c2679c46c4dfaaa8819a6f33598ab8d7023b4db65b760b1a2e3d2282c"}
Nov 02 14:02:51.061627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.061602    4492 generic.go:356] "Generic (PLEG): container finished" podID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerID="d5a8677fc45ea182f6bb102b3de1e857ec9445ee417c1cea8a6472b64aee6a49" exitCode=0
Nov 02 14:02:51.061731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.061711    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" event={"ID":"9c32f87e-206f-4ea4-aa83-d16e7f55fce6","Type":"ContainerDied","Data":"d5a8677fc45ea182f6bb102b3de1e857ec9445ee417c1cea8a6472b64aee6a49"}
Nov 02 14:02:51.064706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.064622    4492 generic.go:356] "Generic (PLEG): container finished" podID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerID="659a129de312b9308c9e7200f443979eb7fa4107a455f4458c9cb24485b68858" exitCode=0
Nov 02 14:02:51.064766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.064729    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb" event={"ID":"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44","Type":"ContainerDied","Data":"659a129de312b9308c9e7200f443979eb7fa4107a455f4458c9cb24485b68858"}
Nov 02 14:02:51.070644 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.070618    4492 generic.go:356] "Generic (PLEG): container finished" podID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerID="3072bd3e449a0d63ab21ee930e2895b445ae250b8e9795c6bd17cd3ebda6704b" exitCode=0
Nov 02 14:02:51.070765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.070742    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg" event={"ID":"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf","Type":"ContainerDied","Data":"3072bd3e449a0d63ab21ee930e2895b445ae250b8e9795c6bd17cd3ebda6704b"}
Nov 02 14:02:51.073717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.073561    4492 generic.go:356] "Generic (PLEG): container finished" podID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerID="1ecbdb9327194b696cf645158caa2b3fed2e326fe04578742fbf4d89f2b43522" exitCode=0
Nov 02 14:02:51.073717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.073662    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx" event={"ID":"b6731c93-cd7d-45ef-ac36-7b10270bbfd2","Type":"ContainerDied","Data":"1ecbdb9327194b696cf645158caa2b3fed2e326fe04578742fbf4d89f2b43522"}
Nov 02 14:02:51.078700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.077973    4492 generic.go:356] "Generic (PLEG): container finished" podID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerID="f72e7387526eb0c2193160265643bae85874f9b650335448941cad678a42bd09" exitCode=0
Nov 02 14:02:51.078700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:51.078007    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb" event={"ID":"96a75551-eb75-4769-9eb5-f841ff7cf738","Type":"ContainerDied","Data":"f72e7387526eb0c2193160265643bae85874f9b650335448941cad678a42bd09"}
Nov 02 14:02:59.184729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:59.184689    4492 generic.go:356] "Generic (PLEG): container finished" podID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerID="3a6bd27c40580ffd94208d862c792ec6954349f51ff83a6556a539c52d19c54a" exitCode=0
Nov 02 14:02:59.185456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:02:59.185417    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx" event={"ID":"b6731c93-cd7d-45ef-ac36-7b10270bbfd2","Type":"ContainerDied","Data":"3a6bd27c40580ffd94208d862c792ec6954349f51ff83a6556a539c52d19c54a"}
Nov 02 14:03:05.258072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:05.257502    4492 generic.go:356] "Generic (PLEG): container finished" podID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerID="8bda0f86f49d6b81d8052a7215249c0c95ad15babe1c067dd5d037ea59288950" exitCode=0
Nov 02 14:03:05.258072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:05.257643    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t" event={"ID":"bc1638ae-03ea-4a0f-98fc-efa4d7305faa","Type":"ContainerDied","Data":"8bda0f86f49d6b81d8052a7215249c0c95ad15babe1c067dd5d037ea59288950"}
Nov 02 14:03:05.304440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:05.304394    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:06.505187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.504819    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:03:06.624002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.623909    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj\" (UniqueName: \"kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj\") pod \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") "
Nov 02 14:03:06.624002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.623999    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util\") pod \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") "
Nov 02 14:03:06.624246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.624050    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle\") pod \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\" (UID: \"bc1638ae-03ea-4a0f-98fc-efa4d7305faa\") "
Nov 02 14:03:06.625295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.625266    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle" pod "bc1638ae-03ea-4a0f-98fc-efa4d7305faa" (UID: "bc1638ae-03ea-4a0f-98fc-efa4d7305faa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:06.629301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.629237    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj" pod "bc1638ae-03ea-4a0f-98fc-efa4d7305faa" (UID: "bc1638ae-03ea-4a0f-98fc-efa4d7305faa"). InnerVolumeSpecName "kube-api-access-dv5qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:06.631337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.631290    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util" pod "bc1638ae-03ea-4a0f-98fc-efa4d7305faa" (UID: "bc1638ae-03ea-4a0f-98fc-efa4d7305faa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:06.725288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.725247    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv5qj\" (UniqueName: \"kubernetes.io/projected/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-kube-api-access-dv5qj\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:06.725288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.725287    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:06.725288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:06.725298    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc1638ae-03ea-4a0f-98fc-efa4d7305faa-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:07.279375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:07.276120    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t" event={"ID":"bc1638ae-03ea-4a0f-98fc-efa4d7305faa","Type":"ContainerDied","Data":"4f7a71404f847c3c7fc2749f2421e96684fb621d58d590c043a25e43a8faadd3"}
Nov 02 14:03:07.279375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:07.276186    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f7a71404f847c3c7fc2749f2421e96684fb621d58d590c043a25e43a8faadd3"
Nov 02 14:03:07.279375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:07.276274    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716gs9t"
Nov 02 14:03:08.285495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:08.285310    4492 generic.go:356] "Generic (PLEG): container finished" podID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerID="c282bbc65aaf49f2446a9bf48ce50c893efab8c187992cbaac0c69fea5cec087" exitCode=0
Nov 02 14:03:08.285495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:08.285389    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg" event={"ID":"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf","Type":"ContainerDied","Data":"c282bbc65aaf49f2446a9bf48ce50c893efab8c187992cbaac0c69fea5cec087"}
Nov 02 14:03:09.293311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.293228    4492 generic.go:356] "Generic (PLEG): container finished" podID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerID="8238557ea32420e5e25730fb5f417e5abf464c734c0ddbbab14c0cf727c75784" exitCode=0
Nov 02 14:03:09.293745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.293320    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk" event={"ID":"ef6494f5-2f58-436e-b7fe-bb4a36438502","Type":"ContainerDied","Data":"8238557ea32420e5e25730fb5f417e5abf464c734c0ddbbab14c0cf727c75784"}
Nov 02 14:03:09.295543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.295514    4492 generic.go:356] "Generic (PLEG): container finished" podID="0b666de9-372e-4659-902c-e8da861c6258" containerID="80d580849616f6a795aa78c6750f29e1cdbeeb008da77b35ce41b878a3fd54e0" exitCode=0
Nov 02 14:03:09.295672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.295611    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn" event={"ID":"0b666de9-372e-4659-902c-e8da861c6258","Type":"ContainerDied","Data":"80d580849616f6a795aa78c6750f29e1cdbeeb008da77b35ce41b878a3fd54e0"}
Nov 02 14:03:09.297830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.297806    4492 generic.go:356] "Generic (PLEG): container finished" podID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerID="497623ce50f01e4327945a0a70ad8d14c431dc4863ac6554ad70ff01db09e80c" exitCode=0
Nov 02 14:03:09.297930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.297891    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx" event={"ID":"03286bc1-501b-4c3b-b464-90ea17aeb465","Type":"ContainerDied","Data":"497623ce50f01e4327945a0a70ad8d14c431dc4863ac6554ad70ff01db09e80c"}
Nov 02 14:03:09.478144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.478109    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:03:09.653046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.653020    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util\") pod \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") "
Nov 02 14:03:09.653144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.653087    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc\" (UniqueName: \"kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc\") pod \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") "
Nov 02 14:03:09.653182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.653141    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle\") pod \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\" (UID: \"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf\") "
Nov 02 14:03:09.653662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.653637    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle" pod "9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" (UID: "9debe8bf-90e1-493f-aeb8-7c9984bcbcaf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:09.655199 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.655178    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc" pod "9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" (UID: "9debe8bf-90e1-493f-aeb8-7c9984bcbcaf"). InnerVolumeSpecName "kube-api-access-lzvrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:09.658821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.658799    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util" pod "9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" (UID: "9debe8bf-90e1-493f-aeb8-7c9984bcbcaf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:09.754481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.754442    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvrc\" (UniqueName: \"kubernetes.io/projected/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-kube-api-access-lzvrc\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:09.754481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.754480    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:09.754660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:09.754495    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9debe8bf-90e1-493f-aeb8-7c9984bcbcaf-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.304750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.304711    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" event={"ID":"9c32f87e-206f-4ea4-aa83-d16e7f55fce6","Type":"ContainerStarted","Data":"f1857a004fb7c8d320fe3a86572fdb6ebea15b6a22a3c23e0a7515a6c874d9ad"}
Nov 02 14:03:10.306727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.306700    4492 generic.go:356] "Generic (PLEG): container finished" podID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerID="39fc70fd8221cb7c0d9fca15f5dfd2afe2b4f5eaf2bfab3f06f8183b57867015" exitCode=0
Nov 02 14:03:10.306847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.306767    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb" event={"ID":"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44","Type":"ContainerDied","Data":"39fc70fd8221cb7c0d9fca15f5dfd2afe2b4f5eaf2bfab3f06f8183b57867015"}
Nov 02 14:03:10.308439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.308421    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg"
Nov 02 14:03:10.308538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.308455    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78g5gg" event={"ID":"9debe8bf-90e1-493f-aeb8-7c9984bcbcaf","Type":"ContainerDied","Data":"0b56d9828f5c17390c19ab41ad2e9c6ecfa715c9054a73a87c21c977dba0b872"}
Nov 02 14:03:10.308538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.308493    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b56d9828f5c17390c19ab41ad2e9c6ecfa715c9054a73a87c21c977dba0b872"
Nov 02 14:03:10.310634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.310613    4492 generic.go:356] "Generic (PLEG): container finished" podID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerID="dd5ea0ca7e046465feda05169fb741bc66d9a3af8cba711913b81adca03243ca" exitCode=0
Nov 02 14:03:10.310734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.310694    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx" event={"ID":"b6731c93-cd7d-45ef-ac36-7b10270bbfd2","Type":"ContainerDied","Data":"dd5ea0ca7e046465feda05169fb741bc66d9a3af8cba711913b81adca03243ca"}
Nov 02 14:03:10.318233 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.318206    4492 generic.go:356] "Generic (PLEG): container finished" podID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerID="68ac3eb19ad1bec804a0d5aa7a2493cda98a4ecb0f08d195421bef73b47d6d54" exitCode=0
Nov 02 14:03:10.318550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.318531    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb" event={"ID":"96a75551-eb75-4769-9eb5-f841ff7cf738","Type":"ContainerDied","Data":"68ac3eb19ad1bec804a0d5aa7a2493cda98a4ecb0f08d195421bef73b47d6d54"}
Nov 02 14:03:10.346991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.345104    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" podStartSLOduration=3.27228664 podStartE2EDuration="35.345086379s" totalImagesPullingTime="32.072799739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:10.345007458 +0000 UTC m=+354.888723052" watchObservedRunningTime="2026-11-02 14:03:10.345086379 +0000 UTC m=+354.888801974"
Nov 02 14:03:10.490793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.490760    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:03:10.492979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.492957    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:03:10.498934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.498914    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:03:10.663988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.663902    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle\") pod \"0b666de9-372e-4659-902c-e8da861c6258\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") "
Nov 02 14:03:10.663988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.663956    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util\") pod \"0b666de9-372e-4659-902c-e8da861c6258\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") "
Nov 02 14:03:10.663988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.663986    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util\") pod \"ef6494f5-2f58-436e-b7fe-bb4a36438502\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") "
Nov 02 14:03:10.664266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664002    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util\") pod \"03286bc1-501b-4c3b-b464-90ea17aeb465\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") "
Nov 02 14:03:10.664266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664037    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5\" (UniqueName: \"kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5\") pod \"03286bc1-501b-4c3b-b464-90ea17aeb465\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") "
Nov 02 14:03:10.664266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664071    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle\") pod \"ef6494f5-2f58-436e-b7fe-bb4a36438502\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") "
Nov 02 14:03:10.664266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664095    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle\") pod \"03286bc1-501b-4c3b-b464-90ea17aeb465\" (UID: \"03286bc1-501b-4c3b-b464-90ea17aeb465\") "
Nov 02 14:03:10.664266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664137    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9\" (UniqueName: \"kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9\") pod \"ef6494f5-2f58-436e-b7fe-bb4a36438502\" (UID: \"ef6494f5-2f58-436e-b7fe-bb4a36438502\") "
Nov 02 14:03:10.664266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664207    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6\" (UniqueName: \"kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6\") pod \"0b666de9-372e-4659-902c-e8da861c6258\" (UID: \"0b666de9-372e-4659-902c-e8da861c6258\") "
Nov 02 14:03:10.664599 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664545    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle" pod "0b666de9-372e-4659-902c-e8da861c6258" (UID: "0b666de9-372e-4659-902c-e8da861c6258"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:10.664684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.664650    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle" pod "ef6494f5-2f58-436e-b7fe-bb4a36438502" (UID: "ef6494f5-2f58-436e-b7fe-bb4a36438502"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:10.665270 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.665241    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle" pod "03286bc1-501b-4c3b-b464-90ea17aeb465" (UID: "03286bc1-501b-4c3b-b464-90ea17aeb465"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:10.666539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.666509    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5" pod "03286bc1-501b-4c3b-b464-90ea17aeb465" (UID: "03286bc1-501b-4c3b-b464-90ea17aeb465"). InnerVolumeSpecName "kube-api-access-clmh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:10.666887 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.666869    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6" pod "0b666de9-372e-4659-902c-e8da861c6258" (UID: "0b666de9-372e-4659-902c-e8da861c6258"). InnerVolumeSpecName "kube-api-access-7kvs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:10.667042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.667025    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9" pod "ef6494f5-2f58-436e-b7fe-bb4a36438502" (UID: "ef6494f5-2f58-436e-b7fe-bb4a36438502"). InnerVolumeSpecName "kube-api-access-9cls9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:10.670702 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.670681    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util" pod "0b666de9-372e-4659-902c-e8da861c6258" (UID: "0b666de9-372e-4659-902c-e8da861c6258"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:10.670805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.670783    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util" pod "ef6494f5-2f58-436e-b7fe-bb4a36438502" (UID: "ef6494f5-2f58-436e-b7fe-bb4a36438502"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:10.671228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.671208    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util" pod "03286bc1-501b-4c3b-b464-90ea17aeb465" (UID: "03286bc1-501b-4c3b-b464-90ea17aeb465"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:10.765910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765880    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.765910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765907    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b666de9-372e-4659-902c-e8da861c6258-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.765910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765916    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.766122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765925    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.766122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765933    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clmh5\" (UniqueName: \"kubernetes.io/projected/03286bc1-501b-4c3b-b464-90ea17aeb465-kube-api-access-clmh5\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.766122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765941    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6494f5-2f58-436e-b7fe-bb4a36438502-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.766122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765957    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03286bc1-501b-4c3b-b464-90ea17aeb465-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.766122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765966    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cls9\" (UniqueName: \"kubernetes.io/projected/ef6494f5-2f58-436e-b7fe-bb4a36438502-kube-api-access-9cls9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.766122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:10.765976    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvs6\" (UniqueName: \"kubernetes.io/projected/0b666de9-372e-4659-902c-e8da861c6258-kube-api-access-7kvs6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.324397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.324338    4492 generic.go:356] "Generic (PLEG): container finished" podID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerID="f1857a004fb7c8d320fe3a86572fdb6ebea15b6a22a3c23e0a7515a6c874d9ad" exitCode=0
Nov 02 14:03:11.324397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.324389    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" event={"ID":"9c32f87e-206f-4ea4-aa83-d16e7f55fce6","Type":"ContainerDied","Data":"f1857a004fb7c8d320fe3a86572fdb6ebea15b6a22a3c23e0a7515a6c874d9ad"}
Nov 02 14:03:11.326380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.326338    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx"
Nov 02 14:03:11.326483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.326390    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpqpx" event={"ID":"03286bc1-501b-4c3b-b464-90ea17aeb465","Type":"ContainerDied","Data":"3ceaa7619bfca977326ae4890b0f29bc19a2f6a3cdb0a751295a1f468ca1f851"}
Nov 02 14:03:11.326483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.326430    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ceaa7619bfca977326ae4890b0f29bc19a2f6a3cdb0a751295a1f468ca1f851"
Nov 02 14:03:11.328322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.328295    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk" event={"ID":"ef6494f5-2f58-436e-b7fe-bb4a36438502","Type":"ContainerDied","Data":"b836c060a17e2f1a2693fd5742d744985cdf4600db157a6f7642fb30e5ef7bc1"}
Nov 02 14:03:11.328322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.328314    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ctbzk"
Nov 02 14:03:11.328516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.328338    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b836c060a17e2f1a2693fd5742d744985cdf4600db157a6f7642fb30e5ef7bc1"
Nov 02 14:03:11.330079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.330062    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn"
Nov 02 14:03:11.330171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.330097    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bpcsn" event={"ID":"0b666de9-372e-4659-902c-e8da861c6258","Type":"ContainerDied","Data":"6b716d9e078095e63745aaf02e0ae5676d2971a7206ed0a970279f90862a44ae"}
Nov 02 14:03:11.330171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.330128    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b716d9e078095e63745aaf02e0ae5676d2971a7206ed0a970279f90862a44ae"
Nov 02 14:03:11.492455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.492432    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:03:11.527115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.527093    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:03:11.529988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.529969    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:03:11.672373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672260    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle\") pod \"96a75551-eb75-4769-9eb5-f841ff7cf738\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") "
Nov 02 14:03:11.672373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672297    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r\" (UniqueName: \"kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r\") pod \"96a75551-eb75-4769-9eb5-f841ff7cf738\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") "
Nov 02 14:03:11.672373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672328    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle\") pod \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") "
Nov 02 14:03:11.672656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672383    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle\") pod \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") "
Nov 02 14:03:11.672656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672498    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz\" (UniqueName: \"kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz\") pod \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") "
Nov 02 14:03:11.672656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672562    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm\" (UniqueName: \"kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm\") pod \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") "
Nov 02 14:03:11.672656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672623    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util\") pod \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\" (UID: \"b6731c93-cd7d-45ef-ac36-7b10270bbfd2\") "
Nov 02 14:03:11.672848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672659    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util\") pod \"96a75551-eb75-4769-9eb5-f841ff7cf738\" (UID: \"96a75551-eb75-4769-9eb5-f841ff7cf738\") "
Nov 02 14:03:11.672848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672699    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util\") pod \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\" (UID: \"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44\") "
Nov 02 14:03:11.672848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672791    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle" pod "ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" (UID: "ec1f92d0-ba00-42f1-bf25-9fb2b6978d44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:11.672996 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.672971    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle" pod "96a75551-eb75-4769-9eb5-f841ff7cf738" (UID: "96a75551-eb75-4769-9eb5-f841ff7cf738"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:11.673075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.673053    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.673131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.673082    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.673793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.673768    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle" pod "b6731c93-cd7d-45ef-ac36-7b10270bbfd2" (UID: "b6731c93-cd7d-45ef-ac36-7b10270bbfd2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:11.674901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.674882    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz" pod "b6731c93-cd7d-45ef-ac36-7b10270bbfd2" (UID: "b6731c93-cd7d-45ef-ac36-7b10270bbfd2"). InnerVolumeSpecName "kube-api-access-nspmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:11.675051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.675033    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r" pod "96a75551-eb75-4769-9eb5-f841ff7cf738" (UID: "96a75551-eb75-4769-9eb5-f841ff7cf738"). InnerVolumeSpecName "kube-api-access-5nb8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:11.675108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.675086    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm" pod "ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" (UID: "ec1f92d0-ba00-42f1-bf25-9fb2b6978d44"). InnerVolumeSpecName "kube-api-access-9rzbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:11.679413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.679390    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util" pod "96a75551-eb75-4769-9eb5-f841ff7cf738" (UID: "96a75551-eb75-4769-9eb5-f841ff7cf738"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:11.680029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.680010    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util" pod "b6731c93-cd7d-45ef-ac36-7b10270bbfd2" (UID: "b6731c93-cd7d-45ef-ac36-7b10270bbfd2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:11.680415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.680343    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util" pod "ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" (UID: "ec1f92d0-ba00-42f1-bf25-9fb2b6978d44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:11.773926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773880    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.773926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773921    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspmz\" (UniqueName: \"kubernetes.io/projected/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-kube-api-access-nspmz\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.773926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773932    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzbm\" (UniqueName: \"kubernetes.io/projected/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-kube-api-access-9rzbm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.773926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773941    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6731c93-cd7d-45ef-ac36-7b10270bbfd2-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.774187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773959    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96a75551-eb75-4769-9eb5-f841ff7cf738-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.774187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773966    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec1f92d0-ba00-42f1-bf25-9fb2b6978d44-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.774187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:11.773974    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nb8r\" (UniqueName: \"kubernetes.io/projected/96a75551-eb75-4769-9eb5-f841ff7cf738-kube-api-access-5nb8r\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:12.336001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.335961    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx" event={"ID":"b6731c93-cd7d-45ef-ac36-7b10270bbfd2","Type":"ContainerDied","Data":"71fc1299c99b1d02a9b1295f5e4526afe7e0af4841f773a8ac0d44a9ea20cf5c"}
Nov 02 14:03:12.336510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.335984    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbkxx"
Nov 02 14:03:12.336510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.336047    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71fc1299c99b1d02a9b1295f5e4526afe7e0af4841f773a8ac0d44a9ea20cf5c"
Nov 02 14:03:12.337797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.337773    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb"
Nov 02 14:03:12.337940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.337774    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c48tpb" event={"ID":"96a75551-eb75-4769-9eb5-f841ff7cf738","Type":"ContainerDied","Data":"aae9bf55d267696ea85a5f7fd3abc95d4d4a90c88b22b68fc294db700dc4c915"}
Nov 02 14:03:12.337940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.337888    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aae9bf55d267696ea85a5f7fd3abc95d4d4a90c88b22b68fc294db700dc4c915"
Nov 02 14:03:12.339877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.339856    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb" event={"ID":"ec1f92d0-ba00-42f1-bf25-9fb2b6978d44","Type":"ContainerDied","Data":"7ce36883e7086fc3be54ce9720a03a46d194eab8bcc28f01006710dcc210c50f"}
Nov 02 14:03:12.339877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.339873    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fdwfb"
Nov 02 14:03:12.340029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.339895    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ce36883e7086fc3be54ce9720a03a46d194eab8bcc28f01006710dcc210c50f"
Nov 02 14:03:12.468040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.468010    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:03:12.580786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.580747    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5\" (UniqueName: \"kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5\") pod \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") "
Nov 02 14:03:12.580962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.580801    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util\") pod \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") "
Nov 02 14:03:12.580962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.580933    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle\") pod \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\" (UID: \"9c32f87e-206f-4ea4-aa83-d16e7f55fce6\") "
Nov 02 14:03:12.581601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.581570    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle" pod "9c32f87e-206f-4ea4-aa83-d16e7f55fce6" (UID: "9c32f87e-206f-4ea4-aa83-d16e7f55fce6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:12.583047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.583023    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5" pod "9c32f87e-206f-4ea4-aa83-d16e7f55fce6" (UID: "9c32f87e-206f-4ea4-aa83-d16e7f55fce6"). InnerVolumeSpecName "kube-api-access-v2zs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:12.588752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.588694    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util" pod "9c32f87e-206f-4ea4-aa83-d16e7f55fce6" (UID: "9c32f87e-206f-4ea4-aa83-d16e7f55fce6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:12.681683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.681646    4492 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-bundle\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:12.681683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.681677    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zs5\" (UniqueName: \"kubernetes.io/projected/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-kube-api-access-v2zs5\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:12.681683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:12.681688    4492 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c32f87e-206f-4ea4-aa83-d16e7f55fce6-util\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:03:13.345652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:13.345610    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p" event={"ID":"9c32f87e-206f-4ea4-aa83-d16e7f55fce6","Type":"ContainerDied","Data":"53437d1ce014a4ff34620b825f5e0d92da256b762aaea9d084acae25fd6619e9"}
Nov 02 14:03:13.346054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:13.345662    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53437d1ce014a4ff34620b825f5e0d92da256b762aaea9d084acae25fd6619e9"
Nov 02 14:03:13.346054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:13.345626    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqk8p"
Nov 02 14:03:15.106420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:15.106390    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:15.221509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:15.221478    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:15.335089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:15.335060    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:16.024746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.024711    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:16.187286 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187250    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"]
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187579    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="pull"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187591    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="pull"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187601    4492 cpu_manager.go:414] "containerMap: removing container" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187606    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187612    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187617    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187623    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187627    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187638    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="pull"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187643    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="pull"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187650    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="util"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187655    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="util"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187659    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="util"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187663    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="util"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187668    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="util"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187672    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="util"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187681    4492 cpu_manager.go:414] "containerMap: removing container" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="pull"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187685    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="pull"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187692    4492 cpu_manager.go:414] "containerMap: removing container" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187696    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="extract"
Nov 02 14:03:16.187693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187703    4492 cpu_manager.go:414] "containerMap: removing container" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187707    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187713    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187717    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187723    4492 cpu_manager.go:414] "containerMap: removing container" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187727    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187735    4492 cpu_manager.go:414] "containerMap: removing container" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187739    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187745    4492 cpu_manager.go:414] "containerMap: removing container" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187749    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187754    4492 cpu_manager.go:414] "containerMap: removing container" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187758    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187764    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187769    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187775    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187778    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187786    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187789    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187795    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187798    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="util"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187804    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187808    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187813    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187818    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187822    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187826    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="pull"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187831    4492 cpu_manager.go:414] "containerMap: removing container" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187835    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187841    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187845    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187852    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187856    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187862    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187866    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187907    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="9debe8bf-90e1-493f-aeb8-7c9984bcbcaf" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187916    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a75551-eb75-4769-9eb5-f841ff7cf738" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187922    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c32f87e-206f-4ea4-aa83-d16e7f55fce6" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187928    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6494f5-2f58-436e-b7fe-bb4a36438502" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187933    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1f92d0-ba00-42f1-bf25-9fb2b6978d44" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187938    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6731c93-cd7d-45ef-ac36-7b10270bbfd2" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187942    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="03286bc1-501b-4c3b-b464-90ea17aeb465" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187947    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b666de9-372e-4659-902c-e8da861c6258" containerName="extract"
Nov 02 14:03:16.188256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.187954    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1638ae-03ea-4a0f-98fc-efa4d7305faa" containerName="extract"
Nov 02 14:03:16.192230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.192212    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.220026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.219991    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 14:03:16.220202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.220047    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 14:03:16.229414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.229388    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 14:03:16.229568 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.229440    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 14:03:16.229568 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.229456    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sd8v9\""
Nov 02 14:03:16.229568 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.229504    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 14:03:16.256287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.256257    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"]
Nov 02 14:03:16.309643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.309547    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsp9\" (UniqueName: \"kubernetes.io/projected/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-kube-api-access-kqsp9\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.309643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.309598    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.309859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.309683    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.309859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.309761    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.309859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.309805    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.309859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.309829    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410298 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410263    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410298 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410309    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410337    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410404    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsp9\" (UniqueName: \"kubernetes.io/projected/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-kube-api-access-kqsp9\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410441    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410471    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.410886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.410845    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.412902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.412871    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.413118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.413097    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.413194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.413124    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.413194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.413161    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.435124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.435091    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsp9\" (UniqueName: \"kubernetes.io/projected/59380b0c-d328-4fa4-b17d-ec9d3ea382fd-kube-api-access-kqsp9\") pod \"opendatahub-operator-controller-manager-78cdb466dc-fxs5r\" (UID: \"59380b0c-d328-4fa4-b17d-ec9d3ea382fd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.437146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.437106    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:16.507593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.507559    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:16.516998 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:03:16.516958    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:03:16.556759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.556711    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:16.686876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.686850    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:16.800600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.800564    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:16.864482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:16.864395    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"]
Nov 02 14:03:16.868912 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:16.868884    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59380b0c_d328_4fa4_b17d_ec9d3ea382fd.slice/crio-8618d7cb59d4438fa5ed7971b147cab2849a6f1a4c9856d02718f92608211c6d WatchSource:0}: Error finding container 8618d7cb59d4438fa5ed7971b147cab2849a6f1a4c9856d02718f92608211c6d: Status 404 returned error can't find the container with id 8618d7cb59d4438fa5ed7971b147cab2849a6f1a4c9856d02718f92608211c6d
Nov 02 14:03:17.363605 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:17.363565    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r" event={"ID":"59380b0c-d328-4fa4-b17d-ec9d3ea382fd","Type":"ContainerStarted","Data":"8618d7cb59d4438fa5ed7971b147cab2849a6f1a4c9856d02718f92608211c6d"}
Nov 02 14:03:18.492705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:18.492670    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:19.143968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:19.143922    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:20.381278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:20.381239    4492 generic.go:356] "Generic (PLEG): container finished" podID="59380b0c-d328-4fa4-b17d-ec9d3ea382fd" containerID="89cc6cb3edf7c481ac3ccaeb23080e2380883b00eb744ab82db195acfe9cf704" exitCode=0
Nov 02 14:03:20.381789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:20.381329    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r" event={"ID":"59380b0c-d328-4fa4-b17d-ec9d3ea382fd","Type":"ContainerDied","Data":"89cc6cb3edf7c481ac3ccaeb23080e2380883b00eb744ab82db195acfe9cf704"}
Nov 02 14:03:20.593116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:20.593032    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:21.387455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:21.387399    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r" event={"ID":"59380b0c-d328-4fa4-b17d-ec9d3ea382fd","Type":"ContainerStarted","Data":"08002cf00bd410a98558ee18dbcfd6328000609c101ee659e8c968959fc5c423"}
Nov 02 14:03:21.388188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:21.387515    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:22.506499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.506438    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r" podStartSLOduration=2.840748125 podStartE2EDuration="6.506421942s" totalImagesPullingTime="3.206444824s" totalInitContainerRuntime="459.228993ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:21.512102637 +0000 UTC m=+366.055818232" watchObservedRunningTime="2026-11-02 14:03:22.506421942 +0000 UTC m=+367.050137580"
Nov 02 14:03:22.507281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.507259    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"]
Nov 02 14:03:22.510706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.510686    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.533333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.533304    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vnnbh\""
Nov 02 14:03:22.533333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.533310    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 14:03:22.533553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.533317    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:03:22.535162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.535138    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"]
Nov 02 14:03:22.554665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.554634    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58p9t\" (UniqueName: \"kubernetes.io/projected/e3a96ea5-a108-42ee-afef-20083d1eb335-kube-api-access-58p9t\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d\" (UID: \"e3a96ea5-a108-42ee-afef-20083d1eb335\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.554788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.554727    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a96ea5-a108-42ee-afef-20083d1eb335-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d\" (UID: \"e3a96ea5-a108-42ee-afef-20083d1eb335\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.655503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.655463    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58p9t\" (UniqueName: \"kubernetes.io/projected/e3a96ea5-a108-42ee-afef-20083d1eb335-kube-api-access-58p9t\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d\" (UID: \"e3a96ea5-a108-42ee-afef-20083d1eb335\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.655650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.655520    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a96ea5-a108-42ee-afef-20083d1eb335-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d\" (UID: \"e3a96ea5-a108-42ee-afef-20083d1eb335\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.655913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.655896    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a96ea5-a108-42ee-afef-20083d1eb335-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d\" (UID: \"e3a96ea5-a108-42ee-afef-20083d1eb335\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.681211 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.681177    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58p9t\" (UniqueName: \"kubernetes.io/projected/e3a96ea5-a108-42ee-afef-20083d1eb335-kube-api-access-58p9t\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d\" (UID: \"e3a96ea5-a108-42ee-afef-20083d1eb335\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:22.822059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:22.821964    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"
Nov 02 14:03:23.159644 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:23.159552    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:23.220406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:23.220345    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d"]
Nov 02 14:03:23.237004 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:23.236976    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a96ea5_a108_42ee_afef_20083d1eb335.slice/crio-62be74dec8f4d1bd9778216d2372caa90d0efb9f63ab424040492f46c412d0cd WatchSource:0}: Error finding container 62be74dec8f4d1bd9778216d2372caa90d0efb9f63ab424040492f46c412d0cd: Status 404 returned error can't find the container with id 62be74dec8f4d1bd9778216d2372caa90d0efb9f63ab424040492f46c412d0cd
Nov 02 14:03:23.398471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:23.398431    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d" event={"ID":"e3a96ea5-a108-42ee-afef-20083d1eb335","Type":"ContainerStarted","Data":"62be74dec8f4d1bd9778216d2372caa90d0efb9f63ab424040492f46c412d0cd"}
Nov 02 14:03:24.297853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:24.297814    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:24.998187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:24.998148    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:25.410777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:25.410690    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d" event={"ID":"e3a96ea5-a108-42ee-afef-20083d1eb335","Type":"ContainerStarted","Data":"7d5fd5d1ff9aea18f26e607b57371eec494f8ecc44ab506ed950115e8df6a517"}
Nov 02 14:03:25.473399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:25.473331    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mdj7d" podStartSLOduration=1.643044017 podStartE2EDuration="3.47331364s" totalImagesPullingTime="1.830269623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:25.463284504 +0000 UTC m=+370.007000097" watchObservedRunningTime="2026-11-02 14:03:25.47331364 +0000 UTC m=+370.017029235"
Nov 02 14:03:25.494832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:25.494798    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:26.348724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:26.348694    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:26.481980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:26.481940    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:27.319266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:27.319229    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:27.467372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:27.467321    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:29.601734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:29.601705    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:29.943673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:29.943628    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:30.217417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.217322    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"]
Nov 02 14:03:30.220632 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.220616    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.234386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.234343    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"]
Nov 02 14:03:30.242279 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.242255    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 14:03:30.242434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.242257    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 14:03:30.242434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.242284    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85kp5\""
Nov 02 14:03:30.304847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.304818    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"]
Nov 02 14:03:30.308377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.308336    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.309651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.309622    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6jkfj\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.309826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.309809    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxb7\" (UniqueName: \"kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6jkfj\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.330455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.330425    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"]
Nov 02 14:03:30.333475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.333454    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rclrg\""
Nov 02 14:03:30.334083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.334069    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 14:03:30.334163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.334125    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 14:03:30.410329 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.410293    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1ea6aa-3e78-43fa-897e-e6498cf8c676-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvdx4\" (UID: \"3b1ea6aa-3e78-43fa-897e-e6498cf8c676\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.410522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.410376    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6jkfj\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.410522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.410454    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpgwh\" (UniqueName: \"kubernetes.io/projected/3b1ea6aa-3e78-43fa-897e-e6498cf8c676-kube-api-access-dpgwh\") pod \"cert-manager-cainjector-5c57bbd4fc-lvdx4\" (UID: \"3b1ea6aa-3e78-43fa-897e-e6498cf8c676\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.410522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.410494    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxb7\" (UniqueName: \"kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6jkfj\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.410745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.410727    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6jkfj\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.433243 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.433211    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxb7\" (UniqueName: \"kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6jkfj\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.511301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.511214    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpgwh\" (UniqueName: \"kubernetes.io/projected/3b1ea6aa-3e78-43fa-897e-e6498cf8c676-kube-api-access-dpgwh\") pod \"cert-manager-cainjector-5c57bbd4fc-lvdx4\" (UID: \"3b1ea6aa-3e78-43fa-897e-e6498cf8c676\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.511301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.511279    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1ea6aa-3e78-43fa-897e-e6498cf8c676-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvdx4\" (UID: \"3b1ea6aa-3e78-43fa-897e-e6498cf8c676\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.532898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.532878    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:30.537221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.537197    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b1ea6aa-3e78-43fa-897e-e6498cf8c676-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvdx4\" (UID: \"3b1ea6aa-3e78-43fa-897e-e6498cf8c676\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.543184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.543160    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpgwh\" (UniqueName: \"kubernetes.io/projected/3b1ea6aa-3e78-43fa-897e-e6498cf8c676-kube-api-access-dpgwh\") pod \"cert-manager-cainjector-5c57bbd4fc-lvdx4\" (UID: \"3b1ea6aa-3e78-43fa-897e-e6498cf8c676\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.635899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.635866    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"
Nov 02 14:03:30.861987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.861909    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"]
Nov 02 14:03:30.866242 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:30.866215    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783dd432_04df_4cd5_a97f_61345dccb4a0.slice/crio-878d04ac64149fa4e62f16202258de13cd0375a28e104eaca92a5cebca7f189c WatchSource:0}: Error finding container 878d04ac64149fa4e62f16202258de13cd0375a28e104eaca92a5cebca7f189c: Status 404 returned error can't find the container with id 878d04ac64149fa4e62f16202258de13cd0375a28e104eaca92a5cebca7f189c
Nov 02 14:03:30.932200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:30.932162    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4"]
Nov 02 14:03:30.935985 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:30.935952    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b1ea6aa_3e78_43fa_897e_e6498cf8c676.slice/crio-13b7da03a88c2e86f4f28ac0615df3428b923e967f8db05c4de7157d5c3221c6 WatchSource:0}: Error finding container 13b7da03a88c2e86f4f28ac0615df3428b923e967f8db05c4de7157d5c3221c6: Status 404 returned error can't find the container with id 13b7da03a88c2e86f4f28ac0615df3428b923e967f8db05c4de7157d5c3221c6
Nov 02 14:03:31.439150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:31.439119    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj" event={"ID":"783dd432-04df-4cd5-a97f-61345dccb4a0","Type":"ContainerStarted","Data":"878d04ac64149fa4e62f16202258de13cd0375a28e104eaca92a5cebca7f189c"}
Nov 02 14:03:31.440540 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:31.440498    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4" event={"ID":"3b1ea6aa-3e78-43fa-897e-e6498cf8c676","Type":"ContainerStarted","Data":"13b7da03a88c2e86f4f28ac0615df3428b923e967f8db05c4de7157d5c3221c6"}
Nov 02 14:03:31.698976 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:31.698837    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:32.128492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:32.128410    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:32.395294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:32.395001    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-fxs5r"
Nov 02 14:03:34.127126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:34.127092    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:34.244018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:34.243985    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:34.954084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:34.953117    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"]
Nov 02 14:03:34.956464 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:34.956441    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:34.972846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:34.972825    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 14:03:34.984703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:34.984677    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"]
Nov 02 14:03:35.035447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.035410    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jqfjk\""
Nov 02 14:03:35.050455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.050411    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkwp\" (UniqueName: \"kubernetes.io/projected/28a0d1ac-075a-43b5-8319-02c228b1fa6a-kube-api-access-8bkwp\") pod \"dns-operator-controller-manager-8557cbdb84-jfj67\" (UID: \"28a0d1ac-075a-43b5-8319-02c228b1fa6a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:35.151739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.151699    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkwp\" (UniqueName: \"kubernetes.io/projected/28a0d1ac-075a-43b5-8319-02c228b1fa6a-kube-api-access-8bkwp\") pod \"dns-operator-controller-manager-8557cbdb84-jfj67\" (UID: \"28a0d1ac-075a-43b5-8319-02c228b1fa6a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:35.192558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.192526    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkwp\" (UniqueName: \"kubernetes.io/projected/28a0d1ac-075a-43b5-8319-02c228b1fa6a-kube-api-access-8bkwp\") pod \"dns-operator-controller-manager-8557cbdb84-jfj67\" (UID: \"28a0d1ac-075a-43b5-8319-02c228b1fa6a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:35.272627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.272543    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:35.363678 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.363649    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:35.464407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.464369    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4" event={"ID":"3b1ea6aa-3e78-43fa-897e-e6498cf8c676","Type":"ContainerStarted","Data":"275505c2fb41a95a15d8944d9eb8ea074ea3506c403d9a3a5749c44b3c7c068d"}
Nov 02 14:03:35.466304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.466274    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj" event={"ID":"783dd432-04df-4cd5-a97f-61345dccb4a0","Type":"ContainerStarted","Data":"a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1"}
Nov 02 14:03:35.466465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.466343    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:35.509885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.509822    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvdx4" podStartSLOduration=1.7472028069999999 podStartE2EDuration="5.509803549s" totalImagesPullingTime="3.762600742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:35.507100887 +0000 UTC m=+380.050816480" watchObservedRunningTime="2026-11-02 14:03:35.509803549 +0000 UTC m=+380.053519145"
Nov 02 14:03:35.548577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.548479    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj" podStartSLOduration=1.062358489 podStartE2EDuration="5.54846347s" totalImagesPullingTime="4.486104981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:35.547614304 +0000 UTC m=+380.091329898" watchObservedRunningTime="2026-11-02 14:03:35.54846347 +0000 UTC m=+380.092179065"
Nov 02 14:03:35.622753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.622715    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"]
Nov 02 14:03:35.628150 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:35.628116    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a0d1ac_075a_43b5_8319_02c228b1fa6a.slice/crio-3b2d61dfdbd98116eebf6684d4b76dfdca2c94c2a9ecb433ae85e065520c2b20 WatchSource:0}: Error finding container 3b2d61dfdbd98116eebf6684d4b76dfdca2c94c2a9ecb433ae85e065520c2b20: Status 404 returned error can't find the container with id 3b2d61dfdbd98116eebf6684d4b76dfdca2c94c2a9ecb433ae85e065520c2b20
Nov 02 14:03:35.795957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.795921    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:35.937868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:35.937832    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:36.098295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:36.098253    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:36.475096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:36.474387    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67" event={"ID":"28a0d1ac-075a-43b5-8319-02c228b1fa6a","Type":"ContainerStarted","Data":"3b2d61dfdbd98116eebf6684d4b76dfdca2c94c2a9ecb433ae85e065520c2b20"}
Nov 02 14:03:37.481381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:37.481276    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67" event={"ID":"28a0d1ac-075a-43b5-8319-02c228b1fa6a","Type":"ContainerStarted","Data":"1195c066e28f6e859addd71f5ba6e5ab922e4c4860af28be75bc7ec955627cf0"}
Nov 02 14:03:37.481750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:37.481386    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:37.532313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:37.532258    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67" podStartSLOduration=1.9831014310000001 podStartE2EDuration="3.532241116s" totalImagesPullingTime="1.549139685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:37.531778756 +0000 UTC m=+382.075494351" watchObservedRunningTime="2026-11-02 14:03:37.532241116 +0000 UTC m=+382.075956709"
Nov 02 14:03:37.605901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:37.605873    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:38.657539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:38.657506    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:38.866312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:38.866276    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:39.812463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:39.812431    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"]
Nov 02 14:03:39.816011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:39.815994    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:39.832779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:39.832755    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-htb7d\""
Nov 02 14:03:39.843915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:39.843890    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"]
Nov 02 14:03:39.891459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:39.891426    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m947\" (UniqueName: \"kubernetes.io/projected/c44cd0a2-9688-4f95-8590-7b095ebc93b8-kube-api-access-7m947\") pod \"authorino-operator-7b7bb4bc77-gbpwl\" (UID: \"c44cd0a2-9688-4f95-8590-7b095ebc93b8\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:39.992407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:39.992367    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m947\" (UniqueName: \"kubernetes.io/projected/c44cd0a2-9688-4f95-8590-7b095ebc93b8-kube-api-access-7m947\") pod \"authorino-operator-7b7bb4bc77-gbpwl\" (UID: \"c44cd0a2-9688-4f95-8590-7b095ebc93b8\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:40.014310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:40.014280    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m947\" (UniqueName: \"kubernetes.io/projected/c44cd0a2-9688-4f95-8590-7b095ebc93b8-kube-api-access-7m947\") pod \"authorino-operator-7b7bb4bc77-gbpwl\" (UID: \"c44cd0a2-9688-4f95-8590-7b095ebc93b8\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:40.128657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:40.128574    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:40.434975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:40.434940    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"]
Nov 02 14:03:40.439968 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:40.439943    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44cd0a2_9688_4f95_8590_7b095ebc93b8.slice/crio-9452f2cec96c607d8da4bf17d6a9723b5fdb9db8d0ed66e9160b3b724e06b6ff WatchSource:0}: Error finding container 9452f2cec96c607d8da4bf17d6a9723b5fdb9db8d0ed66e9160b3b724e06b6ff: Status 404 returned error can't find the container with id 9452f2cec96c607d8da4bf17d6a9723b5fdb9db8d0ed66e9160b3b724e06b6ff
Nov 02 14:03:40.496156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:40.496123    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl" event={"ID":"c44cd0a2-9688-4f95-8590-7b095ebc93b8","Type":"ContainerStarted","Data":"9452f2cec96c607d8da4bf17d6a9723b5fdb9db8d0ed66e9160b3b724e06b6ff"}
Nov 02 14:03:40.905639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:40.905535    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:42.343869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.343834    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:42.455892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.455858    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:42.509260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.509220    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl" event={"ID":"c44cd0a2-9688-4f95-8590-7b095ebc93b8","Type":"ContainerStarted","Data":"7bb2272c3df90857f2ccedae9777a901f2d04c64607eed45c53b11f98f34c7b0"}
Nov 02 14:03:42.509459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.509347    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:42.555085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.555021    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl" podStartSLOduration=2.225505295 podStartE2EDuration="3.55500629s" totalImagesPullingTime="1.329500995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:42.552932119 +0000 UTC m=+387.096647713" watchObservedRunningTime="2026-11-02 14:03:42.55500629 +0000 UTC m=+387.098721884"
Nov 02 14:03:42.956373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.956313    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"]
Nov 02 14:03:42.960036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.960014    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:42.973666 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.973644    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j7szs\""
Nov 02 14:03:42.988297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:42.988264    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"]
Nov 02 14:03:43.020947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:43.020912    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5949g\" (UniqueName: \"kubernetes.io/projected/dd12c716-3e42-4130-a442-dc7b3066ee12-kube-api-access-5949g\") pod \"limitador-operator-controller-manager-5778bb4449-9dxhn\" (UID: \"dd12c716-3e42-4130-a442-dc7b3066ee12\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:43.121864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:43.121824    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5949g\" (UniqueName: \"kubernetes.io/projected/dd12c716-3e42-4130-a442-dc7b3066ee12-kube-api-access-5949g\") pod \"limitador-operator-controller-manager-5778bb4449-9dxhn\" (UID: \"dd12c716-3e42-4130-a442-dc7b3066ee12\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:43.143959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:43.143916    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5949g\" (UniqueName: \"kubernetes.io/projected/dd12c716-3e42-4130-a442-dc7b3066ee12-kube-api-access-5949g\") pod \"limitador-operator-controller-manager-5778bb4449-9dxhn\" (UID: \"dd12c716-3e42-4130-a442-dc7b3066ee12\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:43.273389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:43.273266    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:43.591108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:43.591037    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"]
Nov 02 14:03:43.594766 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:43.594741    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd12c716_3e42_4130_a442_dc7b3066ee12.slice/crio-91f518d8084f56e3e784705a10e5c48cd0f94b714cc19bd8664e0f50996ffd44 WatchSource:0}: Error finding container 91f518d8084f56e3e784705a10e5c48cd0f94b714cc19bd8664e0f50996ffd44: Status 404 returned error can't find the container with id 91f518d8084f56e3e784705a10e5c48cd0f94b714cc19bd8664e0f50996ffd44
Nov 02 14:03:44.520188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:44.520148    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn" event={"ID":"dd12c716-3e42-4130-a442-dc7b3066ee12","Type":"ContainerStarted","Data":"91f518d8084f56e3e784705a10e5c48cd0f94b714cc19bd8664e0f50996ffd44"}
Nov 02 14:03:44.782057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:44.781964    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:46.476914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.476874    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:46.516959 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:03:46.516924    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:03:46.533911 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.533876    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn" event={"ID":"dd12c716-3e42-4130-a442-dc7b3066ee12","Type":"ContainerStarted","Data":"20c702c3adba91ab63bbc5437f2907f10ea9b223a574d634b7d975cc2003f3b0"}
Nov 02 14:03:46.534092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.533984    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:46.573631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.573587    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn" podStartSLOduration=2.506371732 podStartE2EDuration="4.573573658s" totalImagesPullingTime="2.067201926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:46.572369115 +0000 UTC m=+391.116084721" watchObservedRunningTime="2026-11-02 14:03:46.573573658 +0000 UTC m=+391.117289251"
Nov 02 14:03:46.915553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.915442    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"]
Nov 02 14:03:46.919175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.919152    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:46.949808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.949779    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"]
Nov 02 14:03:46.953476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.953436    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:46.955458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.955432    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4flwn\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:46.955557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:46.955467    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2tw\" (UniqueName: \"kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw\") pod \"kuadrant-operator-controller-manager-678d87766f-4flwn\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:47.056704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.056662    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4flwn\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:47.056704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.056701    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2tw\" (UniqueName: \"kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw\") pod \"kuadrant-operator-controller-manager-678d87766f-4flwn\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:47.057062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.057038    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4flwn\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:47.079610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.079581    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2tw\" (UniqueName: \"kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw\") pod \"kuadrant-operator-controller-manager-678d87766f-4flwn\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:47.088050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.088022    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:47.232382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.232310    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:47.340928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.340894    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:47.452231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.452198    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:47.561059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.560961    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"]
Nov 02 14:03:47.578617 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:47.578584    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3918932b_be47_42ad_8685_9bc895afe0f3.slice/crio-d30dcf5cf7118a11e07d47aeddbc891428a6f5371c0bba9c180cc23d87d923a1 WatchSource:0}: Error finding container d30dcf5cf7118a11e07d47aeddbc891428a6f5371c0bba9c180cc23d87d923a1: Status 404 returned error can't find the container with id d30dcf5cf7118a11e07d47aeddbc891428a6f5371c0bba9c180cc23d87d923a1
Nov 02 14:03:47.578949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:47.578929    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:48.488249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:48.488219    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jfj67"
Nov 02 14:03:48.545517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:48.545489    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn" event={"ID":"3918932b-be47-42ad-8685-9bc895afe0f3","Type":"ContainerStarted","Data":"e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d"}
Nov 02 14:03:48.545731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:48.545539    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn" event={"ID":"3918932b-be47-42ad-8685-9bc895afe0f3","Type":"ContainerStarted","Data":"d30dcf5cf7118a11e07d47aeddbc891428a6f5371c0bba9c180cc23d87d923a1"}
Nov 02 14:03:48.545731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:48.545595    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:48.600410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:48.598974    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn" podStartSLOduration=2.598953449 podStartE2EDuration="2.598953449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:48.593868331 +0000 UTC m=+393.137583922" watchObservedRunningTime="2026-11-02 14:03:48.598953449 +0000 UTC m=+393.142669043"
Nov 02 14:03:52.438712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:52.438679    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:52.566787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:52.566752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:52.857422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:52.857331    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:53.274552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.274515    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:53.404289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.404247    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:53.456309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.456279    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"]
Nov 02 14:03:53.464755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.464725    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.485522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.485495    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"]
Nov 02 14:03:53.499488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.499457    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7ksjg\""
Nov 02 14:03:53.515198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.515161    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:53.515892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.515870    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbpwl"
Nov 02 14:03:53.518591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518563    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518594    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518621    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518734    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518788    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr4p\" (UniqueName: \"kubernetes.io/projected/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-kube-api-access-bwr4p\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518812    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518842    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.518971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518958    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.519004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.518986    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.619973    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620099    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620121    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620138    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620166    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620192    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620223    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620255    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr4p\" (UniqueName: \"kubernetes.io/projected/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-kube-api-access-bwr4p\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620286    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620562    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620641    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620786    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620856    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.620895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.620864    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.622775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.622750    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.623025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.623006    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.629748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.629726    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:53.643784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.643758    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.644000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.643976    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr4p\" (UniqueName: \"kubernetes.io/projected/139c2d0e-6d58-405e-a9ad-fc667dcbeaaf-kube-api-access-bwr4p\") pod \"data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8\" (UID: \"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.778676 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.778640    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:53.918786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:53.918677    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:54.059121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.059064    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:54.111241 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.111201    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"]
Nov 02 14:03:54.115959 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:03:54.115930    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139c2d0e_6d58_405e_a9ad_fc667dcbeaaf.slice/crio-42a17b31229645177f41f437fc24140c2c32f070ccc9d9e441eef484400aa4a5 WatchSource:0}: Error finding container 42a17b31229645177f41f437fc24140c2c32f070ccc9d9e441eef484400aa4a5: Status 404 returned error can't find the container with id 42a17b31229645177f41f437fc24140c2c32f070ccc9d9e441eef484400aa4a5
Nov 02 14:03:54.118561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.118530    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:03:54.118641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.118596    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:03:54.118641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.118627    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:03:54.578962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.578922    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8" event={"ID":"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf","Type":"ContainerStarted","Data":"76ac6978c3b5d00c544dda6d330294b16a87e2ab01115a7cd04a4979db6300d2"}
Nov 02 14:03:54.579380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.578967    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8" event={"ID":"139c2d0e-6d58-405e-a9ad-fc667dcbeaaf","Type":"ContainerStarted","Data":"42a17b31229645177f41f437fc24140c2c32f070ccc9d9e441eef484400aa4a5"}
Nov 02 14:03:54.623316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.623267    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8" podStartSLOduration=1.623251762 podStartE2EDuration="1.623251762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:54.620893027 +0000 UTC m=+399.164608621" watchObservedRunningTime="2026-11-02 14:03:54.623251762 +0000 UTC m=+399.166967355"
Nov 02 14:03:54.779515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.779483    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:54.963661 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:54.963590    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:55.784550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:55.784522    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:56.193611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:56.193567    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:56.589774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:56.589679    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:56.590840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:56.590812    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8"
Nov 02 14:03:57.287424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:57.287393    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:57.398124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:57.398076    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:03:57.541623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:57.541547    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9dxhn"
Nov 02 14:03:59.552826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.552798    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:03:59.644189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.644155    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"]
Nov 02 14:03:59.644513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.644483    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj" podUID="783dd432-04df-4cd5-a97f-61345dccb4a0" containerName="manager" containerID="cri-o://a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1" gracePeriod=10
Nov 02 14:03:59.891412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.891386    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:03:59.982420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.982389    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume\") pod \"783dd432-04df-4cd5-a97f-61345dccb4a0\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") "
Nov 02 14:03:59.982597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.982438    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7\" (UniqueName: \"kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7\") pod \"783dd432-04df-4cd5-a97f-61345dccb4a0\" (UID: \"783dd432-04df-4cd5-a97f-61345dccb4a0\") "
Nov 02 14:03:59.982994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.982967    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume" pod "783dd432-04df-4cd5-a97f-61345dccb4a0" (UID: "783dd432-04df-4cd5-a97f-61345dccb4a0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:59.985002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:03:59.984974    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7" pod "783dd432-04df-4cd5-a97f-61345dccb4a0" (UID: "783dd432-04df-4cd5-a97f-61345dccb4a0"). InnerVolumeSpecName "kube-api-access-2pxb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:00.083682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.083605    4492 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/783dd432-04df-4cd5-a97f-61345dccb4a0-extensions-socket-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:04:00.083682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.083631    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pxb7\" (UniqueName: \"kubernetes.io/projected/783dd432-04df-4cd5-a97f-61345dccb4a0-kube-api-access-2pxb7\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:04:00.180229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.180191    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:00.313530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.313498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:00.612061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.612029    4492 generic.go:356] "Generic (PLEG): container finished" podID="783dd432-04df-4cd5-a97f-61345dccb4a0" containerID="a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1" exitCode=0
Nov 02 14:04:00.612479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.612079    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"
Nov 02 14:04:00.612479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.612117    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj" event={"ID":"783dd432-04df-4cd5-a97f-61345dccb4a0","Type":"ContainerDied","Data":"a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1"}
Nov 02 14:04:00.612479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.612167    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj" event={"ID":"783dd432-04df-4cd5-a97f-61345dccb4a0","Type":"ContainerDied","Data":"878d04ac64149fa4e62f16202258de13cd0375a28e104eaca92a5cebca7f189c"}
Nov 02 14:04:00.612479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.612191    4492 scope.go:122] "RemoveContainer" containerID="a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1"
Nov 02 14:04:00.621578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.621558    4492 scope.go:122] "RemoveContainer" containerID="a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1"
Nov 02 14:04:00.621856 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:04:00.621838    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1\": container with ID starting with a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1 not found: ID does not exist" containerID="a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1"
Nov 02 14:04:00.621934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.621869    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1"} err="failed to get container status \"a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1\": rpc error: code = NotFound desc = could not find container \"a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1\": container with ID starting with a8b98858d72f11d73b57469720a56b2c417f2ba3dce7160482bf0558c340cdb1 not found: ID does not exist"
Nov 02 14:04:00.671867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.671838    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"]
Nov 02 14:04:00.683787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:00.683754    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6jkfj"]
Nov 02 14:04:01.578949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:01.578917    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783dd432-04df-4cd5-a97f-61345dccb4a0" path="/var/lib/kubelet/pods/783dd432-04df-4cd5-a97f-61345dccb4a0/volumes"
Nov 02 14:04:02.213677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.213636    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"]
Nov 02 14:04:02.214260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.214213    4492 cpu_manager.go:414] "containerMap: removing container" podUID="783dd432-04df-4cd5-a97f-61345dccb4a0" containerName="manager"
Nov 02 14:04:02.214260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.214229    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783dd432-04df-4cd5-a97f-61345dccb4a0" containerName="manager"
Nov 02 14:04:02.214384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.214302    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="783dd432-04df-4cd5-a97f-61345dccb4a0" containerName="manager"
Nov 02 14:04:02.218889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.218864    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.233896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.233859    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-pnxnv\""
Nov 02 14:04:02.234023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.233907    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 14:04:02.242517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.242492    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 14:04:02.262268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.262239    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 14:04:02.266739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.266719    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"]
Nov 02 14:04:02.305705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.305668    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe8ffd-d93b-42a0-a727-88d122676cbc-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.305886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.305710    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1fe8ffd-d93b-42a0-a727-88d122676cbc-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.305886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.305807    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gqp\" (UniqueName: \"kubernetes.io/projected/a1fe8ffd-d93b-42a0-a727-88d122676cbc-kube-api-access-r7gqp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.305886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.305844    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1fe8ffd-d93b-42a0-a727-88d122676cbc-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.406693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.406659    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe8ffd-d93b-42a0-a727-88d122676cbc-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.406693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.406699    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1fe8ffd-d93b-42a0-a727-88d122676cbc-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.406909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.406732    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gqp\" (UniqueName: \"kubernetes.io/projected/a1fe8ffd-d93b-42a0-a727-88d122676cbc-kube-api-access-r7gqp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.406909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.406772    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1fe8ffd-d93b-42a0-a727-88d122676cbc-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.407231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.407205    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1fe8ffd-d93b-42a0-a727-88d122676cbc-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.407263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.407211    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1fe8ffd-d93b-42a0-a727-88d122676cbc-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.407630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.407611    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1fe8ffd-d93b-42a0-a727-88d122676cbc-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.429339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.429301    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gqp\" (UniqueName: \"kubernetes.io/projected/a1fe8ffd-d93b-42a0-a727-88d122676cbc-kube-api-access-r7gqp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p5lx\" (UID: \"a1fe8ffd-d93b-42a0-a727-88d122676cbc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.531920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.531816    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:02.850015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:02.849882    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"]
Nov 02 14:04:02.855525 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:04:02.855497    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fe8ffd_d93b_42a0_a727_88d122676cbc.slice/crio-b0f960d0174b228058e152e0f839604529f3ed4b8c267ffdd7cbb8ef189c5669 WatchSource:0}: Error finding container b0f960d0174b228058e152e0f839604529f3ed4b8c267ffdd7cbb8ef189c5669: Status 404 returned error can't find the container with id b0f960d0174b228058e152e0f839604529f3ed4b8c267ffdd7cbb8ef189c5669
Nov 02 14:04:03.132598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:03.132512    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:03.252303 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:03.252258    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:03.633126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:03.633085    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx" event={"ID":"a1fe8ffd-d93b-42a0-a727-88d122676cbc","Type":"ContainerStarted","Data":"b0f960d0174b228058e152e0f839604529f3ed4b8c267ffdd7cbb8ef189c5669"}
Nov 02 14:04:05.409874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:05.409786    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:05.646125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:05.646093    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx" event={"ID":"a1fe8ffd-d93b-42a0-a727-88d122676cbc","Type":"ContainerStarted","Data":"7f511ce7224417af0299ed1cc70859e695221286f5e03a674569380d424ada97"}
Nov 02 14:04:05.646276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:05.646170    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:05.695807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:05.695743    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx" podStartSLOduration=1.397793607 podStartE2EDuration="3.695727412s" totalImagesPullingTime="2.297933805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:05.694181431 +0000 UTC m=+410.237897025" watchObservedRunningTime="2026-11-02 14:04:05.695727412 +0000 UTC m=+410.239443045"
Nov 02 14:04:08.025181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:08.025145    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:08.136673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:08.136642    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:14.524645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:14.524610    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:14.644590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:14.644556    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:15.447677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:15.447643    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:16.516669 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:04:16.516631    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:04:20.436046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:20.436004    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:20.546800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:20.546743    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:20.712631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:20.712533    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:20.834030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:20.833992    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:20.989919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:20.989819    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:21.170986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.170952    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:21.394769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.394647    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:21.838995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.838952    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sn9rq"]
Nov 02 14:04:21.847816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.847785    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:21.871413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.871381    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sn9rq"]
Nov 02 14:04:21.874076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.874044    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k46c\" (UniqueName: \"kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c\") pod \"authorino-57f7f8659f-sn9rq\" (UID: \"261aa06d-d677-47a5-b362-1c341d5f1a28\") " pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:21.921737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.921707    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5hbj7\""
Nov 02 14:04:21.935730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.935696    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:21.974765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.974733    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k46c\" (UniqueName: \"kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c\") pod \"authorino-57f7f8659f-sn9rq\" (UID: \"261aa06d-d677-47a5-b362-1c341d5f1a28\") " pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:21.996751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:21.996715    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k46c\" (UniqueName: \"kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c\") pod \"authorino-57f7f8659f-sn9rq\" (UID: \"261aa06d-d677-47a5-b362-1c341d5f1a28\") " pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:22.048002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.047963    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:22.162434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.162307    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:22.457979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.457942    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sn9rq"]
Nov 02 14:04:22.462451 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:04:22.462409    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261aa06d_d677_47a5_b362_1c341d5f1a28.slice/crio-d4e3ec49e0e9d33ba7a48bdf743101896d8291fda459bdead9f3abc0cde62250 WatchSource:0}: Error finding container d4e3ec49e0e9d33ba7a48bdf743101896d8291fda459bdead9f3abc0cde62250: Status 404 returned error can't find the container with id d4e3ec49e0e9d33ba7a48bdf743101896d8291fda459bdead9f3abc0cde62250
Nov 02 14:04:22.576787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.576752    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7dxb9"]
Nov 02 14:04:22.582521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.582497    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:04:22.598770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.598733    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7dxb9"]
Nov 02 14:04:22.680437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.680398    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlk8n\" (UniqueName: \"kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n\") pod \"authorino-56c6c9ddd5-7dxb9\" (UID: \"dc99e19e-8d75-4f04-b0e5-4af52327cd64\") " pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:04:22.736379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.736288    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sn9rq" event={"ID":"261aa06d-d677-47a5-b362-1c341d5f1a28","Type":"ContainerStarted","Data":"d4e3ec49e0e9d33ba7a48bdf743101896d8291fda459bdead9f3abc0cde62250"}
Nov 02 14:04:22.781769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.781735    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlk8n\" (UniqueName: \"kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n\") pod \"authorino-56c6c9ddd5-7dxb9\" (UID: \"dc99e19e-8d75-4f04-b0e5-4af52327cd64\") " pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:04:22.805237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.805204    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlk8n\" (UniqueName: \"kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n\") pod \"authorino-56c6c9ddd5-7dxb9\" (UID: \"dc99e19e-8d75-4f04-b0e5-4af52327cd64\") " pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:04:22.896331 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:22.896299    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:04:23.192166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:23.192130    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7dxb9"]
Nov 02 14:04:23.196005 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:04:23.195972    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc99e19e_8d75_4f04_b0e5_4af52327cd64.slice/crio-32f97d3721a25670096808b75ac442bbafef5235ae3d6407b8ff0d5f8a9214cd WatchSource:0}: Error finding container 32f97d3721a25670096808b75ac442bbafef5235ae3d6407b8ff0d5f8a9214cd: Status 404 returned error can't find the container with id 32f97d3721a25670096808b75ac442bbafef5235ae3d6407b8ff0d5f8a9214cd
Nov 02 14:04:23.742269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:23.742233    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9" event={"ID":"dc99e19e-8d75-4f04-b0e5-4af52327cd64","Type":"ContainerStarted","Data":"32f97d3721a25670096808b75ac442bbafef5235ae3d6407b8ff0d5f8a9214cd"}
Nov 02 14:04:24.091229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:24.091151    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:24.207612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:24.207581    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:26.652445 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:26.652413    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p5lx"
Nov 02 14:04:26.761157 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:26.761115    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sn9rq" event={"ID":"261aa06d-d677-47a5-b362-1c341d5f1a28","Type":"ContainerStarted","Data":"3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26"}
Nov 02 14:04:26.762684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:26.762654    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9" event={"ID":"dc99e19e-8d75-4f04-b0e5-4af52327cd64","Type":"ContainerStarted","Data":"d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120"}
Nov 02 14:04:26.798858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:26.798804    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-sn9rq" podStartSLOduration=2.485021368 podStartE2EDuration="5.798791452s" totalImagesPullingTime="3.313770084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:26.797135403 +0000 UTC m=+431.340851028" watchObservedRunningTime="2026-11-02 14:04:26.798791452 +0000 UTC m=+431.342507045"
Nov 02 14:04:26.834798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:26.834727    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9" podStartSLOduration=2.199414893 podStartE2EDuration="4.834709487s" totalImagesPullingTime="2.635294594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:26.831740372 +0000 UTC m=+431.375455965" watchObservedRunningTime="2026-11-02 14:04:26.834709487 +0000 UTC m=+431.378425081"
Nov 02 14:04:26.863168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:26.863125    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sn9rq"]
Nov 02 14:04:28.772098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:28.772059    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-sn9rq" podUID="261aa06d-d677-47a5-b362-1c341d5f1a28" containerName="authorino" containerID="cri-o://3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26" gracePeriod=30
Nov 02 14:04:29.025968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.025904    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:29.144031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.143991    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c\" (UniqueName: \"kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c\") pod \"261aa06d-d677-47a5-b362-1c341d5f1a28\" (UID: \"261aa06d-d677-47a5-b362-1c341d5f1a28\") "
Nov 02 14:04:29.146261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.146229    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c" pod "261aa06d-d677-47a5-b362-1c341d5f1a28" (UID: "261aa06d-d677-47a5-b362-1c341d5f1a28"). InnerVolumeSpecName "kube-api-access-4k46c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:29.245321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.245291    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k46c\" (UniqueName: \"kubernetes.io/projected/261aa06d-d677-47a5-b362-1c341d5f1a28-kube-api-access-4k46c\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:04:29.283797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.283700    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:29.393409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.393349    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:29.726573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.726544    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:29.780047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.780013    4492 generic.go:356] "Generic (PLEG): container finished" podID="261aa06d-d677-47a5-b362-1c341d5f1a28" containerID="3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26" exitCode=0
Nov 02 14:04:29.780597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.780063    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-sn9rq"
Nov 02 14:04:29.780597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.780115    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sn9rq" event={"ID":"261aa06d-d677-47a5-b362-1c341d5f1a28","Type":"ContainerDied","Data":"3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26"}
Nov 02 14:04:29.780597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.780169    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-sn9rq" event={"ID":"261aa06d-d677-47a5-b362-1c341d5f1a28","Type":"ContainerDied","Data":"d4e3ec49e0e9d33ba7a48bdf743101896d8291fda459bdead9f3abc0cde62250"}
Nov 02 14:04:29.780597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.780196    4492 scope.go:122] "RemoveContainer" containerID="3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26"
Nov 02 14:04:29.789474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.789455    4492 scope.go:122] "RemoveContainer" containerID="3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26"
Nov 02 14:04:29.789751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:04:29.789727    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26\": container with ID starting with 3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26 not found: ID does not exist" containerID="3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26"
Nov 02 14:04:29.789813 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.789767    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26"} err="failed to get container status \"3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26\": rpc error: code = NotFound desc = could not find container \"3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26\": container with ID starting with 3fd9952f1c89175968759005a77884de72ef86a76102a515416d65bbdab83c26 not found: ID does not exist"
Nov 02 14:04:29.828844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.828799    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sn9rq"]
Nov 02 14:04:29.833740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.833710    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-sn9rq"]
Nov 02 14:04:29.851465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.851381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:29.963082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:29.963050    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:31.584669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:31.584633    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261aa06d-d677-47a5-b362-1c341d5f1a28" path="/var/lib/kubelet/pods/261aa06d-d677-47a5-b362-1c341d5f1a28/volumes"
Nov 02 14:04:35.945333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:35.945298    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:36.058810 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:36.058758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:40.858388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:40.858333    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:40.975110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:40.975067    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:41.691653 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:41.691619    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:42.497404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:42.497347    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:42.614916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:42.614883    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:45.486526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:45.486483    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:45.605460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:45.605424    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:04:46.517444 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:04:46.517404    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:04:58.814232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.814186    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"]
Nov 02 14:04:58.814734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.814649    4492 cpu_manager.go:414] "containerMap: removing container" podUID="261aa06d-d677-47a5-b362-1c341d5f1a28" containerName="authorino"
Nov 02 14:04:58.814734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.814665    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261aa06d-d677-47a5-b362-1c341d5f1a28" containerName="authorino"
Nov 02 14:04:58.814816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.814774    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="261aa06d-d677-47a5-b362-1c341d5f1a28" containerName="authorino"
Nov 02 14:04:58.818064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818037    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818532    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818580    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76ac5b5-6284-4fa9-aac3-fb5162e09484-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818671    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818694    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818740    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818816    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.818896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.818883    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqj6\" (UniqueName: \"kubernetes.io/projected/b76ac5b5-6284-4fa9-aac3-fb5162e09484-kube-api-access-tfqj6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.836311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.836281    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 14:04:58.841700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.841682    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 14:04:58.841802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.841696    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 14:04:58.841802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.841725    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xcrsw\""
Nov 02 14:04:58.853921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.853879    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"]
Nov 02 14:04:58.919226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919186    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.919226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919227    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.919475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919253    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqj6\" (UniqueName: \"kubernetes.io/projected/b76ac5b5-6284-4fa9-aac3-fb5162e09484-kube-api-access-tfqj6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.919475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919293    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.919475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919318    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76ac5b5-6284-4fa9-aac3-fb5162e09484-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.919631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919481    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.919631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.919530    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.920107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.920071    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.921827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.921799    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76ac5b5-6284-4fa9-aac3-fb5162e09484-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.921952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.921935    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.922411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.922392    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.922473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.922392    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b76ac5b5-6284-4fa9-aac3-fb5162e09484-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.953658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.953627    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76ac5b5-6284-4fa9-aac3-fb5162e09484-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:58.954277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:58.954256    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqj6\" (UniqueName: \"kubernetes.io/projected/b76ac5b5-6284-4fa9-aac3-fb5162e09484-kube-api-access-tfqj6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9lb82\" (UID: \"b76ac5b5-6284-4fa9-aac3-fb5162e09484\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:59.130561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:59.130470    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:04:59.475477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:59.475441    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"]
Nov 02 14:04:59.480153 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:04:59.480126    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76ac5b5_6284_4fa9_aac3_fb5162e09484.slice/crio-c5d19c832554a1a74c0eabc9a2970eca01f098077f6d9a8de387c8708450eca9 WatchSource:0}: Error finding container c5d19c832554a1a74c0eabc9a2970eca01f098077f6d9a8de387c8708450eca9: Status 404 returned error can't find the container with id c5d19c832554a1a74c0eabc9a2970eca01f098077f6d9a8de387c8708450eca9
Nov 02 14:04:59.937436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:59.937402    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82" event={"ID":"b76ac5b5-6284-4fa9-aac3-fb5162e09484","Type":"ContainerStarted","Data":"c5d19c832554a1a74c0eabc9a2970eca01f098077f6d9a8de387c8708450eca9"}
Nov 02 14:04:59.995335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:04:59.995309    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.122934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.122905    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.260041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.259902    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.394506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.394473    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.530010 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.529887    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.658810 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.658752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.774304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.774271    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:00.907494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:00.907318    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:01.026953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:01.026877    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:01.140147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:01.140114    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:02.153335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:02.153302    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:05:02.153633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:02.153387    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:05:02.960274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:02.960240    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82" event={"ID":"b76ac5b5-6284-4fa9-aac3-fb5162e09484","Type":"ContainerStarted","Data":"0f9e1edc23e8df1151ac2ff9bca313ad9eb7688003c7ff10f13af03162f4f1a4"}
Nov 02 14:05:02.960500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:02.960338    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:05:03.029417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.029346    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82" podStartSLOduration=2.359062591 podStartE2EDuration="5.029333133s" totalImagesPullingTime="2.670270542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:03.01838669 +0000 UTC m=+467.562102283" watchObservedRunningTime="2026-11-02 14:05:03.029333133 +0000 UTC m=+467.573048727"
Nov 02 14:05:03.055540 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.055504    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:03.165277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.165242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:03.275600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.275524    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:03.385640 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.385601    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:03.502006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.501971    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:03.967934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:03.967903    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9lb82"
Nov 02 14:05:05.531226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:05.531189    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:06.325148 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:06.325112    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:06.435500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:06.435459    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:08.434149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:08.434111    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:08.541565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:08.541537    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:15.612597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:15.612539    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:16.517179 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:05:16.517140    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:05:23.261952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:23.261917    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:23.369613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:23.369579    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:37.621523 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:37.621492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:37.737471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:37.737432    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:39.008222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:39.008190    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:39.136006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:39.135979    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:39.276326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:39.276240    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:46.516948 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:05:46.516904    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:05:50.491475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:50.491438    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:50.623536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:50.623496    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:05:51.214261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:51.214227    4492 generic.go:356] "Generic (PLEG): container finished" podID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerID="3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac" exitCode=0
Nov 02 14:05:51.214497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:51.214298    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" event={"ID":"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa","Type":"ContainerDied","Data":"3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"}
Nov 02 14:05:51.214706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:51.214693    4492 scope.go:122] "RemoveContainer" containerID="3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"
Nov 02 14:05:51.215848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:51.215817    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:05:51.215916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:51.215902    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:05:51.215960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:51.215938    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:05:52.221134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:52.221099    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" event={"ID":"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa","Type":"ContainerStarted","Data":"135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c"}
Nov 02 14:05:52.479484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:52.479390    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:05:52.479625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:52.479592    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-clxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Nov 02 14:05:52.479679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:52.479652    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Nov 02 14:05:53.478874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:53.478836    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-clxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Nov 02 14:05:53.479323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:53.478895    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Nov 02 14:05:54.478988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:54.478956    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-clxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Nov 02 14:05:54.479414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:54.479011    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Nov 02 14:05:55.085240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.085200    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7dxb9"]
Nov 02 14:05:55.265062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.265027    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9" podUID="dc99e19e-8d75-4f04-b0e5-4af52327cd64" containerName="authorino" containerID="cri-o://d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120" gracePeriod=30
Nov 02 14:05:55.478822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.478786    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-clxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Nov 02 14:05:55.478969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.478856    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Nov 02 14:05:55.562011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.561969    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:05:55.681028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.680993    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n\" (UniqueName: \"kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n\") pod \"dc99e19e-8d75-4f04-b0e5-4af52327cd64\" (UID: \"dc99e19e-8d75-4f04-b0e5-4af52327cd64\") "
Nov 02 14:05:55.683293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.683268    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n" pod "dc99e19e-8d75-4f04-b0e5-4af52327cd64" (UID: "dc99e19e-8d75-4f04-b0e5-4af52327cd64"). InnerVolumeSpecName "kube-api-access-wlk8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:55.781745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:55.781716    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlk8n\" (UniqueName: \"kubernetes.io/projected/dc99e19e-8d75-4f04-b0e5-4af52327cd64-kube-api-access-wlk8n\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:05:56.271127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.271092    4492 generic.go:356] "Generic (PLEG): container finished" podID="dc99e19e-8d75-4f04-b0e5-4af52327cd64" containerID="d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120" exitCode=0
Nov 02 14:05:56.271313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.271132    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9"
Nov 02 14:05:56.271313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.271153    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9" event={"ID":"dc99e19e-8d75-4f04-b0e5-4af52327cd64","Type":"ContainerDied","Data":"d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120"}
Nov 02 14:05:56.271313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.271185    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7dxb9" event={"ID":"dc99e19e-8d75-4f04-b0e5-4af52327cd64","Type":"ContainerDied","Data":"32f97d3721a25670096808b75ac442bbafef5235ae3d6407b8ff0d5f8a9214cd"}
Nov 02 14:05:56.271313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.271201    4492 scope.go:122] "RemoveContainer" containerID="d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120"
Nov 02 14:05:56.281267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.281247    4492 scope.go:122] "RemoveContainer" containerID="d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120"
Nov 02 14:05:56.281597 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:05:56.281579    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120\": container with ID starting with d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120 not found: ID does not exist" containerID="d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120"
Nov 02 14:05:56.281658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.281608    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120"} err="failed to get container status \"d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120\": rpc error: code = NotFound desc = could not find container \"d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120\": container with ID starting with d8beefcf061bf3c390357c00adfb942c17baf46d36bd65a15d4b18985bca3120 not found: ID does not exist"
Nov 02 14:05:56.316625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.316561    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7dxb9"]
Nov 02 14:05:56.318746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.318717    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7dxb9"]
Nov 02 14:05:56.479320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.479288    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-clxsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Nov 02 14:05:56.479538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:56.479370    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Nov 02 14:05:57.483821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:57.483790    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:05:57.484231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:57.484149    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:05:57.484891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:57.484870    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:05:57.581693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:05:57.580989    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc99e19e-8d75-4f04-b0e5-4af52327cd64" path="/var/lib/kubelet/pods/dc99e19e-8d75-4f04-b0e5-4af52327cd64/volumes"
Nov 02 14:06:05.666135 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:06:05.666093    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:06:15.746485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:06:15.746436    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:06:16.516623 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:06:16.516580    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:06:29.704658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:06:29.704616    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:06:29.917697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:06:29.917658    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:06:34.195876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:06:34.195847    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:06:34.355686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:06:34.355652    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:06:46.516485 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:06:46.516445    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:07:04.139833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:04.139757    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:05.785330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:05.785295    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:14.269051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:14.269014    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:14.788561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:14.788524    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:15.871024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:15.870984    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:16.084245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:16.084209    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:16.205236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:16.205202    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:16.516135 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:07:16.516049    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:07:20.926178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:20.926149    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:23.954591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:23.954556    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:26.036562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.036518    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"]
Nov 02 14:07:26.037071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.037050    4492 cpu_manager.go:414] "containerMap: removing container" podUID="dc99e19e-8d75-4f04-b0e5-4af52327cd64" containerName="authorino"
Nov 02 14:07:26.037134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.037073    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc99e19e-8d75-4f04-b0e5-4af52327cd64" containerName="authorino"
Nov 02 14:07:26.037184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.037178    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc99e19e-8d75-4f04-b0e5-4af52327cd64" containerName="authorino"
Nov 02 14:07:26.040656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.040632    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.063469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.063428    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"]
Nov 02 14:07:26.070321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.070296    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bm7wg\""
Nov 02 14:07:26.070466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.070319    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 14:07:26.090531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.090506    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.090676 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.090575    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.090676 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.090633    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.090676 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.090664    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8qb\" (UniqueName: \"kubernetes.io/projected/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-kube-api-access-6t8qb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.090801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.090695    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.090801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.090730    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.091764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.091744    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:26.120075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.120048    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 14:07:26.120075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.120054    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 14:07:26.191650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.191613    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.191850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.191664    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8qb\" (UniqueName: \"kubernetes.io/projected/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-kube-api-access-6t8qb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.191850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.191704    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.191850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.191744    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.191850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.191815    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.191850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.191838    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.192112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.192084    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.192171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.192147    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.192253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.192232    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.194125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.194101    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.194522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.194507    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.213616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.213589    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8qb\" (UniqueName: \"kubernetes.io/projected/55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9-kube-api-access-6t8qb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4zm5n\" (UID: \"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.358641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.358550    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:26.659214 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.659161    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"]
Nov 02 14:07:26.662028 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:07:26.662001    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dfb5cb_2bbe_4012_a6f0_1b4cc3c439e9.slice/crio-7ccae354793a700b2a42b95815db74642e4f0d60a3b3aeb658a9bca81bcb6378 WatchSource:0}: Error finding container 7ccae354793a700b2a42b95815db74642e4f0d60a3b3aeb658a9bca81bcb6378: Status 404 returned error can't find the container with id 7ccae354793a700b2a42b95815db74642e4f0d60a3b3aeb658a9bca81bcb6378
Nov 02 14:07:26.707922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.707887    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:26.727416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.727377    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n" event={"ID":"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9","Type":"ContainerStarted","Data":"7ccae354793a700b2a42b95815db74642e4f0d60a3b3aeb658a9bca81bcb6378"}
Nov 02 14:07:26.816580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:26.816542    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:28.741654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:28.741611    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n" event={"ID":"55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9","Type":"ContainerStarted","Data":"77251e522e00eefc16789e7af380c48cd4a7e05c0f19ed0ecb9201d06ac57cc1"}
Nov 02 14:07:28.787127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:28.787060    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n" podStartSLOduration=1.612825833 podStartE2EDuration="2.787040411s" totalImagesPullingTime="1.174214578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:28.783680951 +0000 UTC m=+613.327396569" watchObservedRunningTime="2026-11-02 14:07:28.787040411 +0000 UTC m=+613.330756009"
Nov 02 14:07:28.858577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:28.858542    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:29.359775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:29.359736    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:31.020230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.020193    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"]
Nov 02 14:07:31.023971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.023951    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.046449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.046426    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 14:07:31.049013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.048989    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"]
Nov 02 14:07:31.088596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.088569    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:31.149139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.149063    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqm6j\" (UniqueName: \"kubernetes.io/projected/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-kube-api-access-pqm6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.149340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.149155    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.149340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.149187    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.149340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.149203    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.149340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.149222    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.149340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.149269    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.250684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.250642    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqm6j\" (UniqueName: \"kubernetes.io/projected/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-kube-api-access-pqm6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.250856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.250702    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.250856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.250755    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.250856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.250785    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.250856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.250815    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.250856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.250840    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.251293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.251261    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.251443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.251309    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.251443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.251314    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.253512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.253492    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.253636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.253619    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.275696 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.275619    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqm6j\" (UniqueName: \"kubernetes.io/projected/b70b072d-b1a4-4d5c-8745-911dbd0aef1b-kube-api-access-pqm6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw\" (UID: \"b70b072d-b1a4-4d5c-8745-911dbd0aef1b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.338420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.338390    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:31.649858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.649764    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"]
Nov 02 14:07:31.653794 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:07:31.653764    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70b072d_b1a4_4d5c_8745_911dbd0aef1b.slice/crio-3aa065f0ad4a81adfb9ce7149031a608a5398251b3a5022085bb371f05883403 WatchSource:0}: Error finding container 3aa065f0ad4a81adfb9ce7149031a608a5398251b3a5022085bb371f05883403: Status 404 returned error can't find the container with id 3aa065f0ad4a81adfb9ce7149031a608a5398251b3a5022085bb371f05883403
Nov 02 14:07:31.655183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.655164    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:07:31.759799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:31.759762    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw" event={"ID":"b70b072d-b1a4-4d5c-8745-911dbd0aef1b","Type":"ContainerStarted","Data":"3aa065f0ad4a81adfb9ce7149031a608a5398251b3a5022085bb371f05883403"}
Nov 02 14:07:32.633167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:32.633128    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:32.765941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:32.765900    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw" event={"ID":"b70b072d-b1a4-4d5c-8745-911dbd0aef1b","Type":"ContainerStarted","Data":"93f2f70a647125d3fcb3c750fa0f2c41994bebade27d4bd842dd0ad9bdbbf69c"}
Nov 02 14:07:32.822282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:32.822234    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw" podStartSLOduration=2.595360127 podStartE2EDuration="2.822217576s" totalImagesPullingTime="226.857449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:32.820915023 +0000 UTC m=+617.364630611" watchObservedRunningTime="2026-11-02 14:07:32.822217576 +0000 UTC m=+617.365933169"
Nov 02 14:07:33.338836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:33.338788    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:34.290498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.290456    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"]
Nov 02 14:07:34.294736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.294695    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.316427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.316398    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"]
Nov 02 14:07:34.322643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.322610    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 14:07:34.367303 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.367276    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:34.379691 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.379654    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpk7p\" (UniqueName: \"kubernetes.io/projected/e46cbd5b-ce40-4964-a0c7-264420698f31-kube-api-access-vpk7p\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.379856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.379713    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.379856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.379836    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46cbd5b-ce40-4964-a0c7-264420698f31-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.379974 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.379897    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.379974 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.379943    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.380068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.380003    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.480699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.480667    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpk7p\" (UniqueName: \"kubernetes.io/projected/e46cbd5b-ce40-4964-a0c7-264420698f31-kube-api-access-vpk7p\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.480699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.480707    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.480914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.480841    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46cbd5b-ce40-4964-a0c7-264420698f31-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.480914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.480903    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.480997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.480935    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.480997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.480954    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.481134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.481113    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.481289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.481267    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.481378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.481330    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.484928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.484034    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46cbd5b-ce40-4964-a0c7-264420698f31-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.489509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.489481    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46cbd5b-ce40-4964-a0c7-264420698f31-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.505964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.505936    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpk7p\" (UniqueName: \"kubernetes.io/projected/e46cbd5b-ce40-4964-a0c7-264420698f31-kube-api-access-vpk7p\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf\" (UID: \"e46cbd5b-ce40-4964-a0c7-264420698f31\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.610374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.610264    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:34.919566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:34.919491    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"]
Nov 02 14:07:34.924839 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:07:34.924808    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46cbd5b_ce40_4964_a0c7_264420698f31.slice/crio-8335da670267e967f9d6e561b0d07d78ed225d42bbb2f952b13f8a5d90236a67 WatchSource:0}: Error finding container 8335da670267e967f9d6e561b0d07d78ed225d42bbb2f952b13f8a5d90236a67: Status 404 returned error can't find the container with id 8335da670267e967f9d6e561b0d07d78ed225d42bbb2f952b13f8a5d90236a67
Nov 02 14:07:35.788212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:35.788172    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf" event={"ID":"e46cbd5b-ce40-4964-a0c7-264420698f31","Type":"ContainerStarted","Data":"da97bc82a3b40f2a773ae29315d262bc63564224462d84173359b47734f06d43"}
Nov 02 14:07:35.788668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:35.788239    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf" event={"ID":"e46cbd5b-ce40-4964-a0c7-264420698f31","Type":"ContainerStarted","Data":"8335da670267e967f9d6e561b0d07d78ed225d42bbb2f952b13f8a5d90236a67"}
Nov 02 14:07:35.832035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:35.831978    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf" podStartSLOduration=1.5943526110000001 podStartE2EDuration="1.831964454s" totalImagesPullingTime="237.611843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:35.830895162 +0000 UTC m=+620.374610805" watchObservedRunningTime="2026-11-02 14:07:35.831964454 +0000 UTC m=+620.375680047"
Nov 02 14:07:36.359571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:36.359526    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:36.373418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:36.373388    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:36.559061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:36.559024    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:36.611380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:36.611290    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:36.804776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:36.804750    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4zm5n"
Nov 02 14:07:41.339593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:41.339548    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:41.352301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:41.352270    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:41.830602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:41.830576    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw"
Nov 02 14:07:44.611280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:44.611238    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:44.623837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:44.623810    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:44.852547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:44.852512    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf"
Nov 02 14:07:46.516643 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:07:46.516605    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:07:51.234121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.234086    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"]
Nov 02 14:07:51.241179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.241158    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.269040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.269012    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"]
Nov 02 14:07:51.349006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.348966    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsbf\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349003    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349029    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349089    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349140    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349282    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349340    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349390    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.349594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.349447    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450441    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450481    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450524    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsbf\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450544    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450567    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450584    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450603    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450647    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.450958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.450931    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.451052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.451029    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.451343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.451316    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.451466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.451345    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.451466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.451385    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.451466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.451391    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.453841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.453813    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.454050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.454027    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.473529 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.473507    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.473714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.473697    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsbf\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf\") pod \"maas-default-gateway-openshift-default-55fbd65575-b8n6c\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.557154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.557077    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:51.876650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.876580    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"]
Nov 02 14:07:51.879895 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:07:51.879867    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb3a350_5224_4483_99c2_e3dbc240d43e.slice/crio-f4e0a71c4440d7260df32818fdb848eb0d3dae59f16f49a8c41bd73b1ca2a761 WatchSource:0}: Error finding container f4e0a71c4440d7260df32818fdb848eb0d3dae59f16f49a8c41bd73b1ca2a761: Status 404 returned error can't find the container with id f4e0a71c4440d7260df32818fdb848eb0d3dae59f16f49a8c41bd73b1ca2a761
Nov 02 14:07:51.882092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.882059    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:07:51.882152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.882120    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:07:51.882188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:51.882151    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:07:52.888130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:52.888093    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" event={"ID":"ebb3a350-5224-4483-99c2-e3dbc240d43e","Type":"ContainerStarted","Data":"dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5"}
Nov 02 14:07:52.888587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:52.888147    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" event={"ID":"ebb3a350-5224-4483-99c2-e3dbc240d43e","Type":"ContainerStarted","Data":"f4e0a71c4440d7260df32818fdb848eb0d3dae59f16f49a8c41bd73b1ca2a761"}
Nov 02 14:07:52.958449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:52.955476    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podStartSLOduration=1.955451117 podStartE2EDuration="1.955451117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:52.942002965 +0000 UTC m=+637.485718560" watchObservedRunningTime="2026-11-02 14:07:52.955451117 +0000 UTC m=+637.499166706"
Nov 02 14:07:53.557589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:53.557552    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:07:53.559418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:53.559216    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 14:07:53.559418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:53.559265    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 14:07:53.963910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:53.963876    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:54.093125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:54.093086    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:54.557834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:54.557790    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 14:07:54.558054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:54.557848    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 14:07:55.559844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:55.559808    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 14:07:55.560391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:55.559869    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 14:07:55.654856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:55.654783    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:55.775665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:55.775635    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:07:56.560833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:56.560790    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 14:07:56.561254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:56.560883    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 14:07:57.557801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:57.557762    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 14:07:57.558118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:57.558088    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 14:07:58.557697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:58.557658    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 14:07:58.558206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:07:58.557711    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 14:08:00.178766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:00.178738    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:08:00.179246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:00.179118    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:08:00.180104 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:00.180079    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:08:05.908767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:05.908701    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:06.296515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.296477    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"]
Nov 02 14:08:06.300151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.300125    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.328151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.328083    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"]
Nov 02 14:08:06.489709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.489662    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.489939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.489715    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.489939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.489866    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.490024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.489976    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.490024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.490014    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.490109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.490030    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.490109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.490067    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64t5n\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.490184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.490118    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.490184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.490164    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.590923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.590820    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.590923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.590885    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.590948    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.590988    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591011    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591049    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64t5n\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591076    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591111    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591160    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591477    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591715    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591861    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591885    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.591949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.591887    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.593310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.593292    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.593670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.593653    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.613334 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.613300    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.613509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.613445    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64t5n\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n\") pod \"maas-default-gateway-openshift-default-55fbd65575-4nbcz\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.616286 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.616265    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:06.940183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.940142    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"]
Nov 02 14:08:06.943641 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:08:06.943616    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c11fbf_0a2c_4de7_a993_5cd1e9427a93.slice/crio-331e5581b1631debaabcfbd4c676af868e0293883f7a1cd3a0c9a463d847265b WatchSource:0}: Error finding container 331e5581b1631debaabcfbd4c676af868e0293883f7a1cd3a0c9a463d847265b: Status 404 returned error can't find the container with id 331e5581b1631debaabcfbd4c676af868e0293883f7a1cd3a0c9a463d847265b
Nov 02 14:08:06.945742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.945708    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:06.945849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.945792    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:06.945849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.945836    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:06.969147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:06.969115    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" event={"ID":"93c11fbf-0a2c-4de7-a993-5cd1e9427a93","Type":"ContainerStarted","Data":"331e5581b1631debaabcfbd4c676af868e0293883f7a1cd3a0c9a463d847265b"}
Nov 02 14:08:07.976054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:07.976019    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" event={"ID":"93c11fbf-0a2c-4de7-a993-5cd1e9427a93","Type":"ContainerStarted","Data":"f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea"}
Nov 02 14:08:08.031001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:08.030942    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podStartSLOduration=2.030926348 podStartE2EDuration="2.030926348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:08.028166274 +0000 UTC m=+652.571881860" watchObservedRunningTime="2026-11-02 14:08:08.030926348 +0000 UTC m=+652.574642016"
Nov 02 14:08:08.616416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:08.616388    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:08.617914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:08.617893    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 14:08:08.617982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:08.617954    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 14:08:09.617480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:09.617442    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 14:08:09.617906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:09.617507    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 14:08:10.617069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:10.617033    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 14:08:10.617250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:10.617087    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 14:08:11.617512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:11.617471    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 14:08:11.617917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:11.617548    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 14:08:12.617405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:12.617339    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 14:08:12.617591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:12.617432    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 14:08:13.617187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:13.617147    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Nov 02 14:08:13.617409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:13.617206    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Nov 02 14:08:15.218651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:15.218611    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:15.219139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:15.218992    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:15.219858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:15.219832    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:08:15.993459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:15.993423    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:16.516271 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:08:16.516239    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:08:21.341506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.341474    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"]
Nov 02 14:08:21.346311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.346292    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.374067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.374032    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"]
Nov 02 14:08:21.454143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454100    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454141    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7nbp\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454165    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454236    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454289    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454377    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454453    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454483    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.454522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.454501    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.555935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.555884    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.555935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.555933    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7nbp\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.555981    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556010    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556040    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556086    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556143    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556489    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556537    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556558    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556641    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556722    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.556800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556779    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.557025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.556939    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.558821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.558795    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.558909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.558795    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.579615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.579081    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.579615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.579555    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7nbp\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp\") pod \"maas-default-gateway-openshift-default-55fbd65575-bxtwk\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.659506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.659404    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:21.971834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.971802    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"]
Nov 02 14:08:21.975098 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:08:21.975069    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa1ff1d_e21b_40d7_8607_3580f21010d4.slice/crio-9211c2dc29b7a28a239cda306e386b4b41ba97d2fcace59856d8819f599f7df0 WatchSource:0}: Error finding container 9211c2dc29b7a28a239cda306e386b4b41ba97d2fcace59856d8819f599f7df0: Status 404 returned error can't find the container with id 9211c2dc29b7a28a239cda306e386b4b41ba97d2fcace59856d8819f599f7df0
Nov 02 14:08:21.977241 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.977210    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:21.977293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.977278    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:21.977328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:21.977307    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:22.059436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:22.059394    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" event={"ID":"1fa1ff1d-e21b-40d7-8607-3580f21010d4","Type":"ContainerStarted","Data":"3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827"}
Nov 02 14:08:22.059586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:22.059467    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" event={"ID":"1fa1ff1d-e21b-40d7-8607-3580f21010d4","Type":"ContainerStarted","Data":"9211c2dc29b7a28a239cda306e386b4b41ba97d2fcace59856d8819f599f7df0"}
Nov 02 14:08:22.109625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:22.109562    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podStartSLOduration=1.109542328 podStartE2EDuration="1.109542328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:22.107679103 +0000 UTC m=+666.651394710" watchObservedRunningTime="2026-11-02 14:08:22.109542328 +0000 UTC m=+666.653257922"
Nov 02 14:08:22.659671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:22.659632    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:23.661302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:23.661263    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 14:08:23.661720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:23.661317    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 14:08:24.660013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:24.659971    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 14:08:24.660215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:24.660043    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 14:08:25.660257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:25.659855    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 14:08:25.660257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:25.659929    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 14:08:26.660667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:26.660577    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 14:08:26.660667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:26.660644    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 14:08:27.659925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:27.659887    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 14:08:27.660106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:27.659944    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 14:08:28.660008 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:28.659950    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 02 14:08:28.660525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:28.660025    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 02 14:08:30.296885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:30.296850    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:30.297585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:30.297558    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:30.298240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:30.298211    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:08:33.040833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:33.040802    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:38.035348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:38.035314    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:46.517006 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:08:46.516969    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:08:51.756617 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:51.756583    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:51.866700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:51.866666    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:52.221256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.221230    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"]
Nov 02 14:08:52.225379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.225345    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.239314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.239294    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c0c28f-openshift-default-dockercfg-z9fnh\""
Nov 02 14:08:52.255724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.255695    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"]
Nov 02 14:08:52.264005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.263973    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:52.357230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357155    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357309    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjhm\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357340    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357374    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357481    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357502    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357537    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357682    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.357924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.357740    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.414322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.414263    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:52.459076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459042    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459087    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459118    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459182    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjhm\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459217    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459408    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459465    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459498    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459537    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459631    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459603    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459790    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.459830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.459798    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.460300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.460273    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.461983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.461957    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.462047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.461996    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.481579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.481525    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.482720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.482702    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjhm\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.537343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.537272    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:52.540465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.540442    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:52.651037 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.650997    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:52.863938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.863862    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"]
Nov 02 14:08:52.867735 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:08:52.867706    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163c67ab_be95_4ae5_af35_1f68185385db.slice/crio-fb39b4d1ebfcf6e46275d7d41abe8bb70a218884eb15e7ec141443575b706b80 WatchSource:0}: Error finding container fb39b4d1ebfcf6e46275d7d41abe8bb70a218884eb15e7ec141443575b706b80: Status 404 returned error can't find the container with id fb39b4d1ebfcf6e46275d7d41abe8bb70a218884eb15e7ec141443575b706b80
Nov 02 14:08:52.870068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.870036    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:52.870139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.870113    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:52.870175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:52.870148    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:08:53.227716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:53.227679    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" event={"ID":"163c67ab-be95-4ae5-af35-1f68185385db","Type":"ContainerStarted","Data":"1c6773568c89387a3b8502e8d98c970e433826980f3466528ba7adf48ae4d975"}
Nov 02 14:08:53.227882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:53.227728    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" event={"ID":"163c67ab-be95-4ae5-af35-1f68185385db","Type":"ContainerStarted","Data":"fb39b4d1ebfcf6e46275d7d41abe8bb70a218884eb15e7ec141443575b706b80"}
Nov 02 14:08:53.272417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:53.272366    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podStartSLOduration=1.2723372 podStartE2EDuration="1.2723372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:53.271281506 +0000 UTC m=+697.814997099" watchObservedRunningTime="2026-11-02 14:08:53.2723372 +0000 UTC m=+697.816052794"
Nov 02 14:08:53.540738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:53.540657    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:53.546416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:53.546394    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:54.233016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:54.232980    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:54.234076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:54.234057    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:08:55.784018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:55.783977    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:55.927213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:55.927185    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:56.506510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:56.506474    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:56.626773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:56.626747    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:56.779712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:56.779622    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:56.903964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:56.903915    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:57.623232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.623196    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:57.764635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.764588    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:57.813616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.813584    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"]
Nov 02 14:08:57.817323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.817305    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:57.850983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.850952    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"]
Nov 02 14:08:57.858482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.858457    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c0c28f\""
Nov 02 14:08:57.858890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.858848    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c0c28f-dockercfg-rd8pf\""
Nov 02 14:08:57.908304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.908226    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"]
Nov 02 14:08:57.911869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.911850    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:57.916636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.916610    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:57.923812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.923791    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9wm\" (UniqueName: \"kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm\") pod \"payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:57.923944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.923927    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:57.936526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:57.936460    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"]
Nov 02 14:08:58.032262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.029711    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9wm\" (UniqueName: \"kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm\") pod \"payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:58.032262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.029813    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thq4\" (UniqueName: \"kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4\") pod \"payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.032262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.029990    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.032262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.030023    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:58.032262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.030996    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:58.056947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.056912    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:58.061333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.061315    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9wm\" (UniqueName: \"kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm\") pod \"payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:58.131157 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.131118    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:08:58.131492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.131470    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.131571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.131559    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thq4\" (UniqueName: \"kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4\") pod \"payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.132231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.132212    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.154281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.154255    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thq4\" (UniqueName: \"kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4\") pod \"payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.225394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.225335    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:08:58.431237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.431201    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"]
Nov 02 14:08:58.434796 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:08:58.434743    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b857731_b51d_4bcf_bdc6_69cfb034f2c5.slice/crio-cd2662eac0436b79214a039a82dfe1695e69dfacaa35154d6367e5d236378802 WatchSource:0}: Error finding container cd2662eac0436b79214a039a82dfe1695e69dfacaa35154d6367e5d236378802: Status 404 returned error can't find the container with id cd2662eac0436b79214a039a82dfe1695e69dfacaa35154d6367e5d236378802
Nov 02 14:08:58.527964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:58.527932    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"]
Nov 02 14:08:58.531007 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:08:58.530979    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262b06a7_8588_40d3_8da0_8f8b643b5fae.slice/crio-0f36cccc6f9cdc68511d1be1546eb661c3b8c46f8e4a80183c9ff51bf8a0fe68 WatchSource:0}: Error finding container 0f36cccc6f9cdc68511d1be1546eb661c3b8c46f8e4a80183c9ff51bf8a0fe68: Status 404 returned error can't find the container with id 0f36cccc6f9cdc68511d1be1546eb661c3b8c46f8e4a80183c9ff51bf8a0fe68
Nov 02 14:08:59.265047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.265006    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" event={"ID":"262b06a7-8588-40d3-8da0-8f8b643b5fae","Type":"ContainerStarted","Data":"0f36cccc6f9cdc68511d1be1546eb661c3b8c46f8e4a80183c9ff51bf8a0fe68"}
Nov 02 14:08:59.266929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.266898    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" event={"ID":"7b857731-b51d-4bcf-bdc6-69cfb034f2c5","Type":"ContainerStarted","Data":"cd2662eac0436b79214a039a82dfe1695e69dfacaa35154d6367e5d236378802"}
Nov 02 14:08:59.489615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.489569    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:59.637940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.637811    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:59.656517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.656484    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"]
Nov 02 14:08:59.662460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.662433    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.680787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.680041    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"]
Nov 02 14:08:59.687096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.686955    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4be645-dockercfg-xd7fp\""
Nov 02 14:08:59.687096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.687064    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4be645\""
Nov 02 14:08:59.713012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.712980    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"]
Nov 02 14:08:59.717118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.717094    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.738986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.738902    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"]
Nov 02 14:08:59.753396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.752597    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96f8\" (UniqueName: \"kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8\") pod \"payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.753851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.753826    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlps\" (UniqueName: \"kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps\") pod \"payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.754081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.754064    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.754312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.754295    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.857178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.857115    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96f8\" (UniqueName: \"kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8\") pod \"payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.857419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.857257    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlps\" (UniqueName: \"kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps\") pod \"payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.857419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.857302    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.857419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.857345    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.858307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.858278    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.858659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.858632    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.870273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.870203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:08:59.879752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.879720    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96f8\" (UniqueName: \"kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8\") pod \"payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:08:59.880659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.880633    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlps\" (UniqueName: \"kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps\") pod \"payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:08:59.994518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:08:59.994481    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:09:00.061742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.061683    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:09:00.326735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.326697    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"]
Nov 02 14:09:00.329795 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:00.329765    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeee856_ad31_49c8_a84b_d9f3588a8127.slice/crio-e503a84ec52f5efda6e778e39ae499b3af3b85676296ded0bdd028205a0a6194 WatchSource:0}: Error finding container e503a84ec52f5efda6e778e39ae499b3af3b85676296ded0bdd028205a0a6194: Status 404 returned error can't find the container with id e503a84ec52f5efda6e778e39ae499b3af3b85676296ded0bdd028205a0a6194
Nov 02 14:09:00.401401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.391037    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"]
Nov 02 14:09:00.401401 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:00.393077    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc413edf3_9fa9_43aa_9e8a_0c8c8e6bb576.slice/crio-3ed15f3b0f0aa2673f10db6f413768c9a77124cc5f7709d7b8f40d90768d41d5 WatchSource:0}: Error finding container 3ed15f3b0f0aa2673f10db6f413768c9a77124cc5f7709d7b8f40d90768d41d5: Status 404 returned error can't find the container with id 3ed15f3b0f0aa2673f10db6f413768c9a77124cc5f7709d7b8f40d90768d41d5
Nov 02 14:09:00.498812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.498775    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:00.607593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.607516    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:00.683380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.683320    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"]
Nov 02 14:09:00.698498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.698466    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:00.708628 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.706913    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"]
Nov 02 14:09:00.711576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.711551    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-555d1e\""
Nov 02 14:09:00.711859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.711824    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-555d1e-dockercfg-9zkr7\""
Nov 02 14:09:00.769420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.767648    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:00.769420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.767791    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9knm\" (UniqueName: \"kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm\") pod \"payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:00.869768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.869665    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:00.869933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.869755    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9knm\" (UniqueName: \"kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm\") pod \"payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:00.870820 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.870797    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:00.893080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:00.893047    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9knm\" (UniqueName: \"kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm\") pod \"payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:01.027087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:01.027054    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:01.286475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:01.286421    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" event={"ID":"fbeee856-ad31-49c8-a84b-d9f3588a8127","Type":"ContainerStarted","Data":"e503a84ec52f5efda6e778e39ae499b3af3b85676296ded0bdd028205a0a6194"}
Nov 02 14:09:01.290159 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:01.290062    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" event={"ID":"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576","Type":"ContainerStarted","Data":"3ed15f3b0f0aa2673f10db6f413768c9a77124cc5f7709d7b8f40d90768d41d5"}
Nov 02 14:09:01.392991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:01.392961    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"]
Nov 02 14:09:01.395820 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:01.395790    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb907231f_8cb0_4bed_86e5_898b4145d948.slice/crio-b22cc349d8b157762750ffdff1e8adfb642e9d019140620a2fbb5c11c095dae1 WatchSource:0}: Error finding container b22cc349d8b157762750ffdff1e8adfb642e9d019140620a2fbb5c11c095dae1: Status 404 returned error can't find the container with id b22cc349d8b157762750ffdff1e8adfb642e9d019140620a2fbb5c11c095dae1
Nov 02 14:09:01.742003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:01.741926    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:02.297744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.297701    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" event={"ID":"b907231f-8cb0-4bed-86e5-898b4145d948","Type":"ContainerStarted","Data":"91397fe69ff0eb20433d2ffebc5068c9ef9d7d90850c9aadceb56a66e1b689e3"}
Nov 02 14:09:02.297860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.297780    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:02.297860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.297803    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" event={"ID":"b907231f-8cb0-4bed-86e5-898b4145d948","Type":"ContainerStarted","Data":"b22cc349d8b157762750ffdff1e8adfb642e9d019140620a2fbb5c11c095dae1"}
Nov 02 14:09:02.299423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.299398    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" event={"ID":"7b857731-b51d-4bcf-bdc6-69cfb034f2c5","Type":"ContainerStarted","Data":"31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6"}
Nov 02 14:09:02.299539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.299463    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:09:02.301078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.300988    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" event={"ID":"262b06a7-8588-40d3-8da0-8f8b643b5fae","Type":"ContainerStarted","Data":"102ad6ce4cfc5acd5daf63f24b22a19dc6fb69829b147d828226ac8806591e8b"}
Nov 02 14:09:02.301143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.301121    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:09:02.347971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.346930    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" podStartSLOduration=1.5765687339999999 podStartE2EDuration="2.346907835s" totalImagesPullingTime="770.339101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:02.336879649 +0000 UTC m=+706.880595246" watchObservedRunningTime="2026-11-02 14:09:02.346907835 +0000 UTC m=+706.890623430"
Nov 02 14:09:02.377371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.377307    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" podStartSLOduration=2.020757583 podStartE2EDuration="5.377287227s" totalImagesPullingTime="3.356529644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:02.37584755 +0000 UTC m=+706.919563150" watchObservedRunningTime="2026-11-02 14:09:02.377287227 +0000 UTC m=+706.921002827"
Nov 02 14:09:02.416240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:02.416184    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" podStartSLOduration=1.935604116 podStartE2EDuration="5.416164048s" totalImagesPullingTime="3.480559932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:02.41279456 +0000 UTC m=+706.956510156" watchObservedRunningTime="2026-11-02 14:09:02.416164048 +0000 UTC m=+706.959879643"
Nov 02 14:09:03.311210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.310455    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" event={"ID":"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576","Type":"ContainerStarted","Data":"d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8"}
Nov 02 14:09:03.311438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.311387    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:09:03.314563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.314534    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" event={"ID":"fbeee856-ad31-49c8-a84b-d9f3588a8127","Type":"ContainerStarted","Data":"3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d"}
Nov 02 14:09:03.407028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.406960    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" podStartSLOduration=2.505708096 podStartE2EDuration="4.406938665s" totalImagesPullingTime="1.901230569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:03.360709351 +0000 UTC m=+707.904424963" watchObservedRunningTime="2026-11-02 14:09:03.406938665 +0000 UTC m=+707.950654259"
Nov 02 14:09:03.407902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.407864    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" podStartSLOduration=2.418383096 podStartE2EDuration="4.407854416s" totalImagesPullingTime="1.98947132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:03.405765786 +0000 UTC m=+707.949481380" watchObservedRunningTime="2026-11-02 14:09:03.407854416 +0000 UTC m=+707.951570011"
Nov 02 14:09:03.492089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.492057    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:03.618466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.617703    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:03.798957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:03.798919    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:04.319863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:04.319827    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:09:05.392811 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:05.392778    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:05.516294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:05.516264    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:05.680528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:05.680503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:06.038439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:06.032870    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:07.243088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:07.243049    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:09.161319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:09.161275    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:10.274546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:10.274499    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:09:10.274964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:10.274580    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:09:10.937717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:10.937682    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:12.385589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.385551    4492 generic.go:356] "Generic (PLEG): container finished" podID="163c67ab-be95-4ae5-af35-1f68185385db" containerID="1c6773568c89387a3b8502e8d98c970e433826980f3466528ba7adf48ae4d975" exitCode=0
Nov 02 14:09:12.386042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.385622    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" event={"ID":"163c67ab-be95-4ae5-af35-1f68185385db","Type":"ContainerDied","Data":"1c6773568c89387a3b8502e8d98c970e433826980f3466528ba7adf48ae4d975"}
Nov 02 14:09:12.386093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.386079    4492 scope.go:122] "RemoveContainer" containerID="1c6773568c89387a3b8502e8d98c970e433826980f3466528ba7adf48ae4d975"
Nov 02 14:09:12.387430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.387402    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:09:12.387511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.387472    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:09:12.387511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.387504    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:09:12.703208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.703167    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"]
Nov 02 14:09:12.707933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.707909    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.722406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.722240    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:09:12.722406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.722290    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:09:12.722406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.722334    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:09:12.722858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.722838    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:09:12.736410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.730069    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"]
Nov 02 14:09:12.758213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.757564    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.758213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.757636    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.758213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.757680    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsn49\" (UniqueName: \"kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.847864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.847828    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:12.858758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.858710    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.871281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.870470    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.871601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.871243    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.871718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.871700    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsn49\" (UniqueName: \"kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.879860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.879826    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:12.915035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:12.908067    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsn49\" (UniqueName: \"kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49\") pod \"maas-api-revoke-sub-64fa535b-z2ltw\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:13.036435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.035541    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:13.331460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.331334    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:09:13.338231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.331818    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:09:13.338231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.332449    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:09:13.455413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.439725    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" event={"ID":"163c67ab-be95-4ae5-af35-1f68185385db","Type":"ContainerStarted","Data":"7b24b865c628c308390e58ce2b12243b12abdb9a826d46af4050ea2cd4c9cd6b"}
Nov 02 14:09:13.540917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.540838    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:09:13.545814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.545495    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:13.545814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.545550    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:13.816587 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:13.804515    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2818582_06cb_4d92_add1_9ee58b523469.slice/crio-1eb218c8ca26ec1ef28d3087cd74c422211262d2f06da1cc9526e3081dddab1c WatchSource:0}: Error finding container 1eb218c8ca26ec1ef28d3087cd74c422211262d2f06da1cc9526e3081dddab1c: Status 404 returned error can't find the container with id 1eb218c8ca26ec1ef28d3087cd74c422211262d2f06da1cc9526e3081dddab1c
Nov 02 14:09:13.818190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:13.818159    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"]
Nov 02 14:09:14.500973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:14.500841    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw" event={"ID":"f2818582-06cb-4d92-add1-9ee58b523469","Type":"ContainerStarted","Data":"1eb218c8ca26ec1ef28d3087cd74c422211262d2f06da1cc9526e3081dddab1c"}
Nov 02 14:09:14.541920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:14.541880    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:14.542224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:14.542176    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:14.898434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:14.897549    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:15.352385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:15.337530    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:09:15.352385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:15.341512    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:09:15.755383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:15.731521    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:15.772501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:15.772452    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:15.950710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:15.950670    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:16.105436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.105030    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:16.239529 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.239493    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-b8n6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:09:16.239804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.239773    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:09:16.275723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.270740    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:16.275723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.272508    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4nbcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:09:16.275723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.275620    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:09:16.347848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.347802    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-bxtwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:09:16.348157 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.348107    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:09:16.458383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.458206    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:16.555234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.554729    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:16.555234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.554791    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:16.603076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.602947    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:16.709854 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:09:16.709724    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:09:16.729219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.729187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:16.952388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:16.950776    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:17.266705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:17.266549    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:17.389234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:17.389203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:17.555432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:17.544036    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:17.555432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:17.544095    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:17.600143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:17.600111    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:18.503261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:18.503220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:18.550197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:18.549747    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:18.550197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:18.549807    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:18.958299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:18.958229    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:19.541860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:19.541821    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 02 14:09:19.542395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:19.541885    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 02 14:09:19.827153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:19.827065    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw" event={"ID":"f2818582-06cb-4d92-add1-9ee58b523469","Type":"ContainerStarted","Data":"4e47828aea9c1fce7dd17e90a4b7a32d54e9dcf706e3fb9c3485b94e77e63c7d"}
Nov 02 14:09:19.864332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:19.864280    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw" podStartSLOduration=2.3925192 podStartE2EDuration="7.864264648s" totalImagesPullingTime="5.471745448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:19.862986493 +0000 UTC m=+724.406702088" watchObservedRunningTime="2026-11-02 14:09:19.864264648 +0000 UTC m=+724.407980430"
Nov 02 14:09:20.546633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:20.546602    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:09:20.547135 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:20.546950    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:09:20.547764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:20.547742    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:09:20.766808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:20.766776    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:20.837395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:20.837267    4492 generic.go:356] "Generic (PLEG): container finished" podID="f2818582-06cb-4d92-add1-9ee58b523469" containerID="4e47828aea9c1fce7dd17e90a4b7a32d54e9dcf706e3fb9c3485b94e77e63c7d" exitCode=0
Nov 02 14:09:20.837578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:20.837404    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw" event={"ID":"f2818582-06cb-4d92-add1-9ee58b523469","Type":"ContainerDied","Data":"4e47828aea9c1fce7dd17e90a4b7a32d54e9dcf706e3fb9c3485b94e77e63c7d"}
Nov 02 14:09:22.004705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.004389    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:22.046875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.046841    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca\") pod \"f2818582-06cb-4d92-add1-9ee58b523469\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") "
Nov 02 14:09:22.047091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.046922    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca\") pod \"f2818582-06cb-4d92-add1-9ee58b523469\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") "
Nov 02 14:09:22.047091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.046994    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49\" (UniqueName: \"kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49\") pod \"f2818582-06cb-4d92-add1-9ee58b523469\" (UID: \"f2818582-06cb-4d92-add1-9ee58b523469\") "
Nov 02 14:09:22.047772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.047718    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca" pod "f2818582-06cb-4d92-add1-9ee58b523469" (UID: "f2818582-06cb-4d92-add1-9ee58b523469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:09:22.055913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.055880    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49" pod "f2818582-06cb-4d92-add1-9ee58b523469" (UID: "f2818582-06cb-4d92-add1-9ee58b523469"). InnerVolumeSpecName "kube-api-access-lsn49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:09:22.074172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.073210    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca" pod "f2818582-06cb-4d92-add1-9ee58b523469" (UID: "f2818582-06cb-4d92-add1-9ee58b523469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:09:22.149073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.148956    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:09:22.149073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.149009    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsn49\" (UniqueName: \"kubernetes.io/projected/f2818582-06cb-4d92-add1-9ee58b523469-kube-api-access-lsn49\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:09:22.149073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.149026    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2818582-06cb-4d92-add1-9ee58b523469-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:09:22.555835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.555801    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:22.851978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.851882    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw" event={"ID":"f2818582-06cb-4d92-add1-9ee58b523469","Type":"ContainerDied","Data":"1eb218c8ca26ec1ef28d3087cd74c422211262d2f06da1cc9526e3081dddab1c"}
Nov 02 14:09:22.851978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.851934    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"
Nov 02 14:09:22.852229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:22.851941    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb218c8ca26ec1ef28d3087cd74c422211262d2f06da1cc9526e3081dddab1c"
Nov 02 14:09:24.663491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:24.663454    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:26.444541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:26.444509    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:27.424763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:27.424728    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:28.318919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:28.318887    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:30.215512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:30.215476    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:31.984202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:31.984164    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:32.106292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:32.106259    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:32.630435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:32.630399    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:32.752188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:32.752152    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:32.864305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:32.864241    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:33.955271 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:33.955236    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:35.769245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:35.769207    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:35.878770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:35.878730    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:37.639039 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:37.639001    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:38.675786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:38.675739    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:39.095347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.095261    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"]
Nov 02 14:09:39.095903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.095888    4492 cpu_manager.go:414] "containerMap: removing container" podUID="f2818582-06cb-4d92-add1-9ee58b523469" containerName="revoke-keys"
Nov 02 14:09:39.095903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.095904    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2818582-06cb-4d92-add1-9ee58b523469" containerName="revoke-keys"
Nov 02 14:09:39.096040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.096030    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2818582-06cb-4d92-add1-9ee58b523469" containerName="revoke-keys"
Nov 02 14:09:39.099867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.099845    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.115930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.115901    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-758617-openshift-default-dockercfg-vjk8k\""
Nov 02 14:09:39.132714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.132678    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"]
Nov 02 14:09:39.157572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.157534    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:39.220078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220040    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220085    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220190    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220288    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220443    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220485    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220608 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220546    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220602    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.220645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.220639    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.272237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.272201    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:39.322963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.322919    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.323251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.323227    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.323447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.323425    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.323611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.323597    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.323721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.323709    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.323877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.323860    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.323991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.323977    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.324100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.324087    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.324238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.324223    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.328380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.325142    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.338374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.334141    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.346447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.342011    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.346675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.344528    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.347003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.346984    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.347792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.347288    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.347916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.347719    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.350928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.350905    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.364725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.364691    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token\") pod \"e2e-inf-a-758617-openshift-default-754657d5bc-vcf88\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") " pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.416797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.416760    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:39.441070 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.441040    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:39.752806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.752775    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"]
Nov 02 14:09:39.757039 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:39.757010    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcec7d06_a5b0_4779_a782_c714554a468d.slice/crio-cf293cca3df98c706874446e0b659dfad51bbdf7091f50b0698793b7722b900f WatchSource:0}: Error finding container cf293cca3df98c706874446e0b659dfad51bbdf7091f50b0698793b7722b900f: Status 404 returned error can't find the container with id cf293cca3df98c706874446e0b659dfad51bbdf7091f50b0698793b7722b900f
Nov 02 14:09:39.759723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.759692    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:09:39.759804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.759762    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:09:39.759804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:39.759790    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:09:40.020539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:40.020444    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" event={"ID":"dcec7d06-a5b0-4779-a782-c714554a468d","Type":"ContainerStarted","Data":"515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"}
Nov 02 14:09:40.020539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:40.020513    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" event={"ID":"dcec7d06-a5b0-4779-a782-c714554a468d","Type":"ContainerStarted","Data":"cf293cca3df98c706874446e0b659dfad51bbdf7091f50b0698793b7722b900f"}
Nov 02 14:09:40.065746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:40.065686    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podStartSLOduration=1.065670505 podStartE2EDuration="1.065670505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:40.065620737 +0000 UTC m=+744.609336331" watchObservedRunningTime="2026-11-02 14:09:40.065670505 +0000 UTC m=+744.609386100"
Nov 02 14:09:40.417505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:40.417411    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:40.422757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:40.422729    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:41.026470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:41.026437    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:41.027507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:41.027488    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:09:41.244473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:41.244439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:42.493667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:42.493638    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:42.636979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:42.636947    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:42.784146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:42.784060    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:43.364373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:43.364337    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:43.878803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:43.878768    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:44.275943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.275899    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:44.385986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.385947    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:44.408308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.408270    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"]
Nov 02 14:09:44.412369 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.412325    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.434701 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.434672    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-758617\""
Nov 02 14:09:44.438868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.438843    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-758617-dockercfg-77vbn\""
Nov 02 14:09:44.439262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.439241    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"]
Nov 02 14:09:44.538081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.536711    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"]
Nov 02 14:09:44.542602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.542573    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.573306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.573276    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"]
Nov 02 14:09:44.581654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.581617    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m\") pod \"payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.581945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.581910    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.683480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.683443    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bml\" (UniqueName: \"kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml\") pod \"payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.683671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.683596    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m\") pod \"payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.683671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.683651    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.683784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.683724    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.684576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.684551    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.706113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.706079    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m\") pod \"payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.727137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.727098    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:44.784952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.784924    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.785076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.784988    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bml\" (UniqueName: \"kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml\") pod \"payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.786063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.786040    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.806978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.806901    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bml\" (UniqueName: \"kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml\") pod \"payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") " pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:44.911174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:44.911133    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:45.034479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:45.034440    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"]
Nov 02 14:09:45.041281 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:45.041242    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86f680c_2f36_47fa_811b_041a12a46fd4.slice/crio-bdaf271d01286c3e2d613ed851149bcb60e30951091a88f074746039fca2789c WatchSource:0}: Error finding container bdaf271d01286c3e2d613ed851149bcb60e30951091a88f074746039fca2789c: Status 404 returned error can't find the container with id bdaf271d01286c3e2d613ed851149bcb60e30951091a88f074746039fca2789c
Nov 02 14:09:45.064441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:45.064338    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" event={"ID":"f86f680c-2f36-47fa-811b-041a12a46fd4","Type":"ContainerStarted","Data":"bdaf271d01286c3e2d613ed851149bcb60e30951091a88f074746039fca2789c"}
Nov 02 14:09:45.211374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:45.211328    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"]
Nov 02 14:09:45.214899 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:45.214869    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ebe684_4231_4d1c_8896_f3e5ab3dbe48.slice/crio-aa4c49ec89cb8ba0d52e4f8bd8ab5cfb7b4491f6d6721b4609c061408079fca7 WatchSource:0}: Error finding container aa4c49ec89cb8ba0d52e4f8bd8ab5cfb7b4491f6d6721b4609c061408079fca7: Status 404 returned error can't find the container with id aa4c49ec89cb8ba0d52e4f8bd8ab5cfb7b4491f6d6721b4609c061408079fca7
Nov 02 14:09:45.239798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:45.239769    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:46.072786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.072748    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" event={"ID":"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48","Type":"ContainerStarted","Data":"58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204"}
Nov 02 14:09:46.073304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.072814    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" event={"ID":"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48","Type":"ContainerStarted","Data":"aa4c49ec89cb8ba0d52e4f8bd8ab5cfb7b4491f6d6721b4609c061408079fca7"}
Nov 02 14:09:46.073304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.072917    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:46.074553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.074530    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" event={"ID":"f86f680c-2f36-47fa-811b-041a12a46fd4","Type":"ContainerStarted","Data":"333fd7470a14a2c62d73f4e614b88375a8b74b2a187d4911be26b7241146be94"}
Nov 02 14:09:46.074688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.074670    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:46.110678 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.110581    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" podStartSLOduration=2.110562435 podStartE2EDuration="2.110562435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:46.108721255 +0000 UTC m=+750.652436850" watchObservedRunningTime="2026-11-02 14:09:46.110562435 +0000 UTC m=+750.654278029"
Nov 02 14:09:46.146100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:46.146040    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" podStartSLOduration=2.14602206 podStartE2EDuration="2.14602206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:46.144251292 +0000 UTC m=+750.687966883" watchObservedRunningTime="2026-11-02 14:09:46.14602206 +0000 UTC m=+750.689737655"
Nov 02 14:09:46.523248 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:09:46.523207    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:09:47.020337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:47.020282    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:48.611144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.611101    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"]
Nov 02 14:09:48.614938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.614918    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.628480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.628449    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.628638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.628526    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.628686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.628633    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.628783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.628764    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xfg\" (UniqueName: \"kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.650614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.650580    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:09:48.650814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.650689    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-758617-serving-cert\""
Nov 02 14:09:48.650814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.650736    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:09:48.650814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.650761    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 14:09:48.651007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.650850    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-245cc\""
Nov 02 14:09:48.656158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.656115    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"]
Nov 02 14:09:48.723825 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.723778    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:48.729326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.729291    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.729601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.729333    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.729601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.729432    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xfg\" (UniqueName: \"kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.729601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.729506    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.729790 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:09:48.729630    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-serving-cert: secret "maas-api-e2e-inf-a-758617-serving-cert" not found
Nov 02 14:09:48.729790 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:09:48.729699    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls podName:ac1a136a-3003-467b-8fb6-70e051b39d73 nodeName:}" failed. No retries permitted until 2026-11-02 14:09:49.229677324 +0000 UTC m=+753.773392901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls") pod "maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" (UID: "ac1a136a-3003-467b-8fb6-70e051b39d73") : secret "maas-api-e2e-inf-a-758617-serving-cert" not found
Nov 02 14:09:48.730222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.730193    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.732951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.732926    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.757175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.756953    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xfg\" (UniqueName: \"kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:48.855380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.854179    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:48.966405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:48.966348    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:49.256682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:49.256585    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:49.259301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:49.259273    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\") pod \"maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:49.538426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:49.535246    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:49.894409 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:09:49.890052    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1a136a_3003_467b_8fb6_70e051b39d73.slice/crio-db0f160aadb4418a435f40bf0f92ed8af98f86aec889e948150f5763aae02cac WatchSource:0}: Error finding container db0f160aadb4418a435f40bf0f92ed8af98f86aec889e948150f5763aae02cac: Status 404 returned error can't find the container with id db0f160aadb4418a435f40bf0f92ed8af98f86aec889e948150f5763aae02cac
Nov 02 14:09:49.894409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:49.890156    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"]
Nov 02 14:09:50.118971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:50.118934    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" event={"ID":"ac1a136a-3003-467b-8fb6-70e051b39d73","Type":"ContainerStarted","Data":"db0f160aadb4418a435f40bf0f92ed8af98f86aec889e948150f5763aae02cac"}
Nov 02 14:09:50.543812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:50.543730    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:52.230687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:52.230646    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:53.141137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:53.141090    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" event={"ID":"ac1a136a-3003-467b-8fb6-70e051b39d73","Type":"ContainerStarted","Data":"a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8"}
Nov 02 14:09:53.141325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:53.141163    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:09:53.183187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:53.182872    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" podStartSLOduration=2.635932488 podStartE2EDuration="5.18285455s" totalImagesPullingTime="2.546922062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:53.179207157 +0000 UTC m=+757.722922752" watchObservedRunningTime="2026-11-02 14:09:53.18285455 +0000 UTC m=+757.726570146"
Nov 02 14:09:53.614945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:53.614913    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:53.953905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:53.953867    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:55.730057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:55.730026    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:57.097707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:57.097676    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:09:57.098195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:57.097751    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:09:57.445990 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:57.445958    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:59.127690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:59.127658    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:09:59.163332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:09:59.163289    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:10:00.843491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:00.843448    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:02.642718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:02.642681    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:03.239869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:03.239833    4492 generic.go:356] "Generic (PLEG): container finished" podID="dcec7d06-a5b0-4779-a782-c714554a468d" containerID="515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c" exitCode=0
Nov 02 14:10:03.240118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:03.239919    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" event={"ID":"dcec7d06-a5b0-4779-a782-c714554a468d","Type":"ContainerDied","Data":"515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"}
Nov 02 14:10:03.240448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:03.240414    4492 scope.go:122] "RemoveContainer" containerID="515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"
Nov 02 14:10:03.241763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:03.241734    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:03.241879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:03.241814    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:03.241879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:03.241865    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:04.062412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:04.053935    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:04.255379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:04.254638    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" event={"ID":"dcec7d06-a5b0-4779-a782-c714554a468d","Type":"ContainerStarted","Data":"9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24"}
Nov 02 14:10:04.417928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:04.417842    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:10:04.418103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:04.418082    4492 patch_prober.go:29] interesting pod/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 14:10:04.418146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:04.418130    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 14:10:04.457996 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:04.457951    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:05.418177 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:05.418141    4492 patch_prober.go:29] interesting pod/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 14:10:05.418618 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:05.418209    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 14:10:06.164557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:06.164525    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:06.274113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:06.274057    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:06.417273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:06.417191    4492 patch_prober.go:29] interesting pod/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 14:10:06.417273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:06.417244    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 14:10:07.417216 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:07.417176    4492 patch_prober.go:29] interesting pod/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 14:10:07.417654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:07.417245    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 14:10:08.050381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:08.043177    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:08.417302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:08.417221    4492 patch_prober.go:29] interesting pod/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 14:10:08.417302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:08.417283    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 14:10:09.561481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:09.561448    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:10:09.562287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:09.562263    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:10:09.564387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:09.564341    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:10:10.175774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:10.175741    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:12.080456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:12.080424    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:12.861206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:12.861172    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:13.367998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.367968    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:13.808734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.808698    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"]
Nov 02 14:10:13.813716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.813690    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.828787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.828763    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-854b9e-openshift-default-dockercfg-xdzsq\""
Nov 02 14:10:13.843152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.843120    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"]
Nov 02 14:10:13.866780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.866744    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.917627    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.917725    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq24h\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.917776    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.917809    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.917903    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.917952    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.918002    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.918036    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.931381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.918083    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:13.996861 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:13.996829    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:14.023027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.022976    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq24h\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023296    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023493    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023678    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023733    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023782    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023814    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023865    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.023903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.023894    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.024854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.024832    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.025301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.024861    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.025453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.025429    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.025688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.025666    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.031409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.026532    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.036380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.035428    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.037117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.037084    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.069075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.068999    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.070075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.070047    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq24h\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h\") pod \"e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") " pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.126555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.126514    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:14.132553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.132513    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:14.475130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.475092    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"]
Nov 02 14:10:14.478506 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:10:14.478469    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0270c6f0_c594_4e27_abc8_2e192ca43cf0.slice/crio-2b7eeb4dfdb7f2660f3eaa3c0706ab2b4069e11ec7992509efe7165ee47336a1 WatchSource:0}: Error finding container 2b7eeb4dfdb7f2660f3eaa3c0706ab2b4069e11ec7992509efe7165ee47336a1: Status 404 returned error can't find the container with id 2b7eeb4dfdb7f2660f3eaa3c0706ab2b4069e11ec7992509efe7165ee47336a1
Nov 02 14:10:14.481065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.481024    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:14.481164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.481114    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:14.481164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:14.481156    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:15.341900 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:15.341862    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" event={"ID":"0270c6f0-c594-4e27-abc8-2e192ca43cf0","Type":"ContainerStarted","Data":"6e8819583d425ab328c4154c866aebd0f394b924410dc17d441d897794cec1fc"}
Nov 02 14:10:15.343080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:15.343045    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" event={"ID":"0270c6f0-c594-4e27-abc8-2e192ca43cf0","Type":"ContainerStarted","Data":"2b7eeb4dfdb7f2660f3eaa3c0706ab2b4069e11ec7992509efe7165ee47336a1"}
Nov 02 14:10:15.395911 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:15.395844    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podStartSLOduration=2.395823354 podStartE2EDuration="2.395823354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:15.383582727 +0000 UTC m=+779.927298324" watchObservedRunningTime="2026-11-02 14:10:15.395823354 +0000 UTC m=+779.939538949"
Nov 02 14:10:15.878699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:15.878670    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:16.133737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:16.133647    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:16.139298 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:16.139272    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:16.347891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:16.347862    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:16.348756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:16.348739    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:16.363969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:16.363938    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:16.524253 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:10:16.524209    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:10:17.822936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:17.822888    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:19.631417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:19.631381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:21.436335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:21.436288    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:22.652132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.652090    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"]
Nov 02 14:10:22.659959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.659939    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.681444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.681412    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-555d1e-openshift-default-dockercfg-s5d5d\""
Nov 02 14:10:22.688464 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.688434    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"]
Nov 02 14:10:22.731865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.731838    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.731874    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.731990    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.732043    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.732106    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.732159    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.732213    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666tf\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.732246    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.732294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.732274    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833530    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833585    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833603    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833662    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833684    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833735    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666tf\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.833779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833775    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833809    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.833839    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.834003    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.834127    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.834306    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.834405    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.834486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.834414    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.836162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.836135    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.836588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.836568    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.855686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.855641    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.855833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.855751    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-666tf\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:22.974507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:22.974470    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:23.267229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:23.267129    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:23.350491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:23.350459    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"]
Nov 02 14:10:23.354398 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:10:23.354333    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f05d9b_6d57_43fc_a8f1_261f676d8678.slice/crio-909529b7b7d489d6d275b1efce03656d25ba5a25013fa61c538fb3d33a36e95e WatchSource:0}: Error finding container 909529b7b7d489d6d275b1efce03656d25ba5a25013fa61c538fb3d33a36e95e: Status 404 returned error can't find the container with id 909529b7b7d489d6d275b1efce03656d25ba5a25013fa61c538fb3d33a36e95e
Nov 02 14:10:23.357325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:23.357290    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:23.357449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:23.357427    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:23.357490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:23.357463    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:23.401151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:23.401117    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" event={"ID":"c0f05d9b-6d57-43fc-a8f1-261f676d8678","Type":"ContainerStarted","Data":"909529b7b7d489d6d275b1efce03656d25ba5a25013fa61c538fb3d33a36e95e"}
Nov 02 14:10:24.409178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:24.409145    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" event={"ID":"c0f05d9b-6d57-43fc-a8f1-261f676d8678","Type":"ContainerStarted","Data":"65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911"}
Nov 02 14:10:24.466370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:24.466293    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podStartSLOduration=2.466278126 podStartE2EDuration="2.466278126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:24.464788726 +0000 UTC m=+789.008504322" watchObservedRunningTime="2026-11-02 14:10:24.466278126 +0000 UTC m=+789.009993719"
Nov 02 14:10:24.975518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:24.975487    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:24.977457 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:24.977424    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 14:10:24.977619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:24.977479    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 14:10:25.042338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:25.042301    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:25.985380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:25.984471    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 14:10:25.985380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:25.984535    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 14:10:26.959019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:26.958983    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:26.975505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:26.975475    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 14:10:26.975661 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:26.975528    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 14:10:27.907249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:27.907214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:27.975504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:27.975470    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 14:10:27.975678 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:27.975536    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 14:10:28.045380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:28.043198    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:28.957039 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:28.957005    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:28.975049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:28.975021    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 14:10:28.975221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:28.975074    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 14:10:29.603231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.603200    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:29.716695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.716664    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:29.790419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.790381    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"]
Nov 02 14:10:29.795971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.795940    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:29.821539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.821333    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-854b9e\""
Nov 02 14:10:29.821539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.821458    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-854b9e-dockercfg-9v5dw\""
Nov 02 14:10:29.826020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.825989    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"]
Nov 02 14:10:29.830082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.830049    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznqk\" (UniqueName: \"kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk\") pod \"payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:29.830214 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.830121    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:29.939488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.939433    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznqk\" (UniqueName: \"kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk\") pod \"payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:29.943662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.943612    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:29.951228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.951183    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:29.985378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:29.984557    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznqk\" (UniqueName: \"kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk\") pod \"payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:30.115585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.115543    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:30.170485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.169574    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:30.170485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.170369    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:30.171891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.171840    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:10:30.432147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.432119    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"]
Nov 02 14:10:30.434792 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:10:30.434761    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e86c391_5506_479e_b9d4_66f8a9b58c68.slice/crio-8a805a3979803f4d9aa3175b781e6e1980a3e3df66934d41da01315f961674fb WatchSource:0}: Error finding container 8a805a3979803f4d9aa3175b781e6e1980a3e3df66934d41da01315f961674fb: Status 404 returned error can't find the container with id 8a805a3979803f4d9aa3175b781e6e1980a3e3df66934d41da01315f961674fb
Nov 02 14:10:30.465869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.465841    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" event={"ID":"5e86c391-5506-479e-b9d4-66f8a9b58c68","Type":"ContainerStarted","Data":"8a805a3979803f4d9aa3175b781e6e1980a3e3df66934d41da01315f961674fb"}
Nov 02 14:10:30.726305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:30.726216    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:31.473713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:31.473673    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" event={"ID":"5e86c391-5506-479e-b9d4-66f8a9b58c68","Type":"ContainerStarted","Data":"89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb"}
Nov 02 14:10:31.474121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:31.473769    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:31.519111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:31.519052    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" podStartSLOduration=2.519035764 podStartE2EDuration="2.519035764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:31.51387246 +0000 UTC m=+796.057588056" watchObservedRunningTime="2026-11-02 14:10:31.519035764 +0000 UTC m=+796.062751357"
Nov 02 14:10:32.629795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:32.629758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:34.432246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:34.432214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:36.090371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.090331    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"]
Nov 02 14:10:36.096827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.096803    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.125399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.118732    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-854b9e-serving-cert\""
Nov 02 14:10:36.128105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.128072    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"]
Nov 02 14:10:36.173739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.173704    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:36.229085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.229041    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.229277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.229117    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.229277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.229218    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58ht\" (UniqueName: \"kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.229447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.229425    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.285065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.285034    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:36.346494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.341279    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.346494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.341397    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.346494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.341440    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58ht\" (UniqueName: \"kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.346494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.341581    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.352414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.349433    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.352414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.352107    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.364830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.358937    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.385953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.377076    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58ht\" (UniqueName: \"kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht\") pod \"maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.397065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.397025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:36.410025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.409993    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:36.713788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:36.713719    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"]
Nov 02 14:10:36.716705 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:10:36.716679    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830deaaf_5d41_4ea7_8019_1db5650b80b0.slice/crio-cde83e23e36c0153125406f8c6dfd27eb03f66c09cd3f22c936fb1b835902bdb WatchSource:0}: Error finding container cde83e23e36c0153125406f8c6dfd27eb03f66c09cd3f22c936fb1b835902bdb: Status 404 returned error can't find the container with id cde83e23e36c0153125406f8c6dfd27eb03f66c09cd3f22c936fb1b835902bdb
Nov 02 14:10:37.526115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:37.526076    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv" event={"ID":"830deaaf-5d41-4ea7-8019-1db5650b80b0","Type":"ContainerStarted","Data":"7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5"}
Nov 02 14:10:37.526589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:37.526127    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv" event={"ID":"830deaaf-5d41-4ea7-8019-1db5650b80b0","Type":"ContainerStarted","Data":"cde83e23e36c0153125406f8c6dfd27eb03f66c09cd3f22c936fb1b835902bdb"}
Nov 02 14:10:37.526589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:37.526184    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:37.564649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:37.564597    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv" podStartSLOduration=1.201410735 podStartE2EDuration="1.564582777s" totalImagesPullingTime="363.172042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:37.563712414 +0000 UTC m=+802.107428009" watchObservedRunningTime="2026-11-02 14:10:37.564582777 +0000 UTC m=+802.108298371"
Nov 02 14:10:38.201869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:38.201837    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:39.400794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:39.400761    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:40.073495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:40.073456    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:41.843556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:41.843520    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:42.482392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:42.482350    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:10:43.566612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:43.566580    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:10:43.670864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:43.670814    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:45.338390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:45.337724    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:46.517597 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:10:46.517560    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:10:47.122667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:47.122624    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:48.972051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:48.972014    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:50.921119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:50.921031    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:51.696244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:51.692725    4492 generic.go:356] "Generic (PLEG): container finished" podID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerID="6e8819583d425ab328c4154c866aebd0f394b924410dc17d441d897794cec1fc" exitCode=0
Nov 02 14:10:51.696244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:51.692901    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" event={"ID":"0270c6f0-c594-4e27-abc8-2e192ca43cf0","Type":"ContainerDied","Data":"6e8819583d425ab328c4154c866aebd0f394b924410dc17d441d897794cec1fc"}
Nov 02 14:10:51.696244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:51.693536    4492 scope.go:122] "RemoveContainer" containerID="6e8819583d425ab328c4154c866aebd0f394b924410dc17d441d897794cec1fc"
Nov 02 14:10:51.710898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:51.709480    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:51.710898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:51.709640    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:51.710898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:51.709728    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:10:52.706379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:52.705378    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" event={"ID":"0270c6f0-c594-4e27-abc8-2e192ca43cf0","Type":"ContainerStarted","Data":"75ed3f58ad0e518324e03db079f4affaa388192e7bd03321c4641c17ffbf8b46"}
Nov 02 14:10:52.759382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:52.756213    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:53.067436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:53.067304    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:53.148381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:53.140755    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:53.148381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:53.140881    4492 patch_prober.go:29] interesting pod/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 14:10:53.148381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:53.140923    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 14:10:54.136804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:54.136766    4492 patch_prober.go:29] interesting pod/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 14:10:54.137459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:54.137427    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 14:10:54.649380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:54.640802    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:55.136834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:55.136796    4492 patch_prober.go:29] interesting pod/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 14:10:55.137488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:55.137456    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 14:10:56.157538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:56.157499    4492 patch_prober.go:29] interesting pod/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 14:10:56.158102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:56.157561    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 14:10:56.451124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:56.451067    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:56.564746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:56.564706    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:10:57.144573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:57.144528    4492 patch_prober.go:29] interesting pod/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 14:10:57.145014 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:57.144968    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 14:10:58.137746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:58.137714    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:58.138152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:58.138001    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:58.138966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:58.138947    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:10:58.366468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:10:58.366437    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:00.145282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:00.145252    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:01.930245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:01.930211    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:03.684791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:03.684710    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:05.534414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:05.534370    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:06.310333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:06.310297    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:07.252598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.252558    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:07.810837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.810795    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"]
Nov 02 14:11:07.815764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.815738    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.851327 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.851295    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"]
Nov 02 14:11:07.935726 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.935693    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:07.960074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960031    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960122    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960149    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960182    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960206    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960246    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960266    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.960308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960288    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:07.961158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:07.960327    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmwt\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.061892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.061796    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.061892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.061866    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.061892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.061885    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.061918    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.061935    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062092    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062133    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062163    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062204    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmwt\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062283    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062329    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062592    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062621    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.062759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.062704    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.064559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.064532    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.064758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.064739    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.084675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.084642    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.085042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.085020    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmwt\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.139015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.138606    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:08.501471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.501445    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"]
Nov 02 14:11:08.504297 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:11:08.504266    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf225768f_8c3d_4184_9cdd_14de3500fbb6.slice/crio-03386391fa783744b7885c445f9e0bf25ce2fd2147c2e94a786ac5be1790fa61 WatchSource:0}: Error finding container 03386391fa783744b7885c445f9e0bf25ce2fd2147c2e94a786ac5be1790fa61: Status 404 returned error can't find the container with id 03386391fa783744b7885c445f9e0bf25ce2fd2147c2e94a786ac5be1790fa61
Nov 02 14:11:08.507187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.507149    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:11:08.507282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.507243    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:11:08.507330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.507308    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:11:08.842521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.842410    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" event={"ID":"f225768f-8c3d-4184-9cdd-14de3500fbb6","Type":"ContainerStarted","Data":"4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78"}
Nov 02 14:11:08.842521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.842479    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" event={"ID":"f225768f-8c3d-4184-9cdd-14de3500fbb6","Type":"ContainerStarted","Data":"03386391fa783744b7885c445f9e0bf25ce2fd2147c2e94a786ac5be1790fa61"}
Nov 02 14:11:08.888244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:08.888184    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podStartSLOduration=1.888167258 podStartE2EDuration="1.888167258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:08.885915686 +0000 UTC m=+833.429631273" watchObservedRunningTime="2026-11-02 14:11:08.888167258 +0000 UTC m=+833.431882852"
Nov 02 14:11:09.036602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:09.036562    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:09.142030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:09.141917    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:09.144430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:09.144037    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:09.144430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:09.144117    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:10.148590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:10.148548    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:10.149224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:10.149189    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:10.964081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:10.964043    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:11.142842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:11.142802    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:11.143153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:11.143121    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:11.707414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:11.679243    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:12.143917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:12.142699    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:12.143917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:12.142770    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:13.066593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:13.066471    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:13.142652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:13.142616    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:13.142917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:13.142891    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:13.875552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:13.875505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:14.141842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:14.141746    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:14.142502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:14.142465    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:14.448392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:14.443212    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:14.566403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:14.559533    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:14.948066 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:14.948034    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:15.140705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:15.140649    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 14:11:15.140891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:15.140723    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 14:11:16.235161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:16.235125    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:16.235775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:16.235578    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:16.237348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:16.237322    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:11:16.486659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:16.486554    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:16.518860 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:11:16.518824    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:11:16.829806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:16.829728    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:18.540574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:18.540540    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:18.672027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:18.671993    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:18.816795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:18.816693    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:20.304153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.304113    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:20.416815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.416782    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:20.488722 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.488691    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"]
Nov 02 14:11:20.493969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.493948    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.516502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.516475    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"]
Nov 02 14:11:20.521577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.521549    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bf0b0d\""
Nov 02 14:11:20.521765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.521626    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bf0b0d-dockercfg-x5rtr\""
Nov 02 14:11:20.554387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.554260    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f\") pod \"payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.554554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.554430    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.641583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.639104    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"]
Nov 02 14:11:20.699009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.689189    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f\") pod \"payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.699504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.699473    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.701260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.701235    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.741409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.728454    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"]
Nov 02 14:11:20.741409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.728578    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:20.743096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.742529    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f\") pod \"payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.743810 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.743780    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:20.810431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.806148    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:20.810431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.806280    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpkh\" (UniqueName: \"kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh\") pod \"payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:20.839134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.839106    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:20.909657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.909241    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpkh\" (UniqueName: \"kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh\") pod \"payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:20.911710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.910425    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:20.911710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.910732    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:20.933004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:20.932950    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpkh\" (UniqueName: \"kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh\") pod \"payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:21.060408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:21.060369    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:21.195110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:21.193698    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"]
Nov 02 14:11:21.199797 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:11:21.196032    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256171e0_9920_4309_bb65_8cee75fac1ed.slice/crio-a53152738acaf68a06d1ac066daa849b15a1d4fcffce3811b21d00d53c0615af WatchSource:0}: Error finding container a53152738acaf68a06d1ac066daa849b15a1d4fcffce3811b21d00d53c0615af: Status 404 returned error can't find the container with id a53152738acaf68a06d1ac066daa849b15a1d4fcffce3811b21d00d53c0615af
Nov 02 14:11:21.594057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:21.593960    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"]
Nov 02 14:11:21.597301 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:11:21.597239    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f4629_f879_4c79_aa08_71cbb4ad864d.slice/crio-6b43cd229d10a9928142d5a5553dc5887557455949fdec18194fa26e6a350b8f WatchSource:0}: Error finding container 6b43cd229d10a9928142d5a5553dc5887557455949fdec18194fa26e6a350b8f: Status 404 returned error can't find the container with id 6b43cd229d10a9928142d5a5553dc5887557455949fdec18194fa26e6a350b8f
Nov 02 14:11:22.019728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.019690    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" event={"ID":"256171e0-9920-4309-bb65-8cee75fac1ed","Type":"ContainerStarted","Data":"fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c"}
Nov 02 14:11:22.019949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.019755    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" event={"ID":"256171e0-9920-4309-bb65-8cee75fac1ed","Type":"ContainerStarted","Data":"a53152738acaf68a06d1ac066daa849b15a1d4fcffce3811b21d00d53c0615af"}
Nov 02 14:11:22.019949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.019777    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:22.021263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.021235    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" event={"ID":"100f4629-f879-4c79-aa08-71cbb4ad864d","Type":"ContainerStarted","Data":"d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a"}
Nov 02 14:11:22.021405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.021297    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:22.021405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.021314    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" event={"ID":"100f4629-f879-4c79-aa08-71cbb4ad864d","Type":"ContainerStarted","Data":"6b43cd229d10a9928142d5a5553dc5887557455949fdec18194fa26e6a350b8f"}
Nov 02 14:11:22.071793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.071737    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" podStartSLOduration=2.071721759 podStartE2EDuration="2.071721759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:22.060324972 +0000 UTC m=+846.604040568" watchObservedRunningTime="2026-11-02 14:11:22.071721759 +0000 UTC m=+846.615437353"
Nov 02 14:11:22.101117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.101061    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" podStartSLOduration=2.101044226 podStartE2EDuration="2.101044226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:22.09791981 +0000 UTC m=+846.641635405" watchObservedRunningTime="2026-11-02 14:11:22.101044226 +0000 UTC m=+846.644759820"
Nov 02 14:11:22.627148 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:22.627115    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:24.440312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:24.440277    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:26.237026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:26.236928    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:28.127444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:28.127406    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:29.806790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.806751    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"]
Nov 02 14:11:29.811146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.811128    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:29.834519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.834468    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-bf0b0d-serving-cert\""
Nov 02 14:11:29.840294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.840260    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"]
Nov 02 14:11:29.892127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.892091    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:29.950245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.950213    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcj76\" (UniqueName: \"kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:29.950430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.950379    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:29.950479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.950435    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:29.950553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:29.950536    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.018563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.018475    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:30.052136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.052102    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.052323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.052156    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.052323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.052219    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.052323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.052265    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcj76\" (UniqueName: \"kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.053064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.053033    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.054871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.054841    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.054977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.054892    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.075158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.075086    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcj76\" (UniqueName: \"kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76\") pod \"maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.126380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.126316    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:30.436586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:30.436559    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"]
Nov 02 14:11:30.439714 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:11:30.439686    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911d6ffa_04a7_4547_b669_4351aa9f2533.slice/crio-07eca7b9753f08f55bce45d2d4289cc2ff0a2c2943f3dbaff39b54b13f2c605c WatchSource:0}: Error finding container 07eca7b9753f08f55bce45d2d4289cc2ff0a2c2943f3dbaff39b54b13f2c605c: Status 404 returned error can't find the container with id 07eca7b9753f08f55bce45d2d4289cc2ff0a2c2943f3dbaff39b54b13f2c605c
Nov 02 14:11:31.085954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:31.085918    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr" event={"ID":"911d6ffa-04a7-4547-b669-4351aa9f2533","Type":"ContainerStarted","Data":"f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4"}
Nov 02 14:11:31.086425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:31.085980    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr" event={"ID":"911d6ffa-04a7-4547-b669-4351aa9f2533","Type":"ContainerStarted","Data":"07eca7b9753f08f55bce45d2d4289cc2ff0a2c2943f3dbaff39b54b13f2c605c"}
Nov 02 14:11:31.086425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:31.086146    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:31.143928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:31.143812    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr" podStartSLOduration=1.728273947 podStartE2EDuration="2.143799058s" totalImagesPullingTime="415.525111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:31.130660469 +0000 UTC m=+855.674376064" watchObservedRunningTime="2026-11-02 14:11:31.143799058 +0000 UTC m=+855.687514651"
Nov 02 14:11:31.819342 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:31.819308    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:33.030680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:33.030647    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:11:33.031133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:33.030733    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:11:33.632774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:33.632743    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:34.860910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:34.860882    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:34.972829 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:34.972793    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:35.426497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:35.426466    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:37.097375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:37.097332    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:11:37.133274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:37.133221    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:38.946698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:38.946665    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:41.004740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:41.004708    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:43.060894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:43.060859    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:45.315606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:45.315571    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:46.520432 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:11:46.520348    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:11:47.817544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:47.817502    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:48.877600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:48.877563    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:49.669536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:49.669507    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:50.371168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:50.371137    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:50.913020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:50.912986    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:51.068546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:51.068493    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:51.646665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:51.646632    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:53.516863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:53.516825    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:54.778683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:54.778648    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:54.939747 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:54.939678    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:55.052796 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:55.052692    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:55.592684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:55.592647    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:56.579525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.579495    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:56.694530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.694496    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:56.747242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.747206    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"]
Nov 02 14:11:56.753144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.753118    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.780240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.780206    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"]
Nov 02 14:11:56.780690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.780665    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2bb4ef\""
Nov 02 14:11:56.781099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.781049    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2bb4ef-dockercfg-xfn99\""
Nov 02 14:11:56.781867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.781837    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48b6j\" (UniqueName: \"kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j\") pod \"payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.781987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.781925    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.887462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.884162    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.887462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.884568    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48b6j\" (UniqueName: \"kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j\") pod \"payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.887462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.885971    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.924129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.924095    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"]
Nov 02 14:11:56.937669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.937399    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48b6j\" (UniqueName: \"kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j\") pod \"payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:56.945695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.945664    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:56.968462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:56.966171    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"]
Nov 02 14:11:57.089757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.089722    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvtc\" (UniqueName: \"kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc\") pod \"payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.089951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.089782    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.092978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.092955    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:57.191619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.191575    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.191819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.191800    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvtc\" (UniqueName: \"kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc\") pod \"payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.192480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.192455    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.218576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.218548    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvtc\" (UniqueName: \"kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc\") pod \"payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.283475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.283426    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:57.405322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.405200    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"]
Nov 02 14:11:57.409065 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:11:57.409033    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e68747_c49e_4767_bc47_f92c0509aecd.slice/crio-02642c97da688adb779992bd3a68287c45ea82baef8b7045939121681d5f9386 WatchSource:0}: Error finding container 02642c97da688adb779992bd3a68287c45ea82baef8b7045939121681d5f9386: Status 404 returned error can't find the container with id 02642c97da688adb779992bd3a68287c45ea82baef8b7045939121681d5f9386
Nov 02 14:11:57.589205 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.589175    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"]
Nov 02 14:11:57.591933 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:11:57.591904    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f4ff8b_716f_4c46_8382_d2c762207fb1.slice/crio-7a8f38f5dfd5adf50797a7ec3cb9a257ad716fe97b0db9c02a1f7c411a5fe1df WatchSource:0}: Error finding container 7a8f38f5dfd5adf50797a7ec3cb9a257ad716fe97b0db9c02a1f7c411a5fe1df: Status 404 returned error can't find the container with id 7a8f38f5dfd5adf50797a7ec3cb9a257ad716fe97b0db9c02a1f7c411a5fe1df
Nov 02 14:11:57.730660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:57.730627    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:11:58.395495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.395455    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" event={"ID":"38e68747-c49e-4767-bc47-f92c0509aecd","Type":"ContainerStarted","Data":"9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea"}
Nov 02 14:11:58.395714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.395592    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" event={"ID":"38e68747-c49e-4767-bc47-f92c0509aecd","Type":"ContainerStarted","Data":"02642c97da688adb779992bd3a68287c45ea82baef8b7045939121681d5f9386"}
Nov 02 14:11:58.396744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.396720    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:11:58.399527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.399327    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" event={"ID":"58f4ff8b-716f-4c46-8382-d2c762207fb1","Type":"ContainerStarted","Data":"eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5"}
Nov 02 14:11:58.399527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.399459    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" event={"ID":"58f4ff8b-716f-4c46-8382-d2c762207fb1","Type":"ContainerStarted","Data":"7a8f38f5dfd5adf50797a7ec3cb9a257ad716fe97b0db9c02a1f7c411a5fe1df"}
Nov 02 14:11:58.399527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.399497    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:11:58.464390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.462167    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" podStartSLOduration=2.46214819 podStartE2EDuration="2.46214819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:58.447796021 +0000 UTC m=+882.991511617" watchObservedRunningTime="2026-11-02 14:11:58.46214819 +0000 UTC m=+883.005863786"
Nov 02 14:11:58.504762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:58.502003    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" podStartSLOduration=2.501982398 podStartE2EDuration="2.501982398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:58.499651828 +0000 UTC m=+883.043367422" watchObservedRunningTime="2026-11-02 14:11:58.501982398 +0000 UTC m=+883.045697993"
Nov 02 14:11:59.934634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:11:59.934600    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:01.745465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:01.745426    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:03.524164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:03.524129    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:05.326571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:05.326535    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:06.491263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:06.491227    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:07.086440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:07.086408    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:07.548691 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:07.548659    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:08.852254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:08.847679    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"]
Nov 02 14:12:08.913634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:08.913303    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:08.941383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:08.940430    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"]
Nov 02 14:12:08.941383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:08.940572    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:08.955595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:08.955529    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2bb4ef-serving-cert\""
Nov 02 14:12:09.032235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.032203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:09.067380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.067160    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.067380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.067229    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.067380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.067268    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.067380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.067306    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2wr\" (UniqueName: \"kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.150208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.150116    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:09.169028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.168997    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.169217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.169039    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.169217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.169073    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.169217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.169118    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2wr\" (UniqueName: \"kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.169874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.169808    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.171707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.171687    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.171829 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.171808    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.190656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.190624    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2wr\" (UniqueName: \"kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr\") pod \"maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.261723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.261686    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:09.411727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.411631    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:12:09.587987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:09.587963    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"]
Nov 02 14:12:09.590581 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:12:09.590555    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785248a9_5836_4d30_a522_0be624cf1b3f.slice/crio-2aa077f5ee09613d5049993ede417d2325d690590c3c9e3acfbe1b06338594cc WatchSource:0}: Error finding container 2aa077f5ee09613d5049993ede417d2325d690590c3c9e3acfbe1b06338594cc: Status 404 returned error can't find the container with id 2aa077f5ee09613d5049993ede417d2325d690590c3c9e3acfbe1b06338594cc
Nov 02 14:12:10.417936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:10.417846    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:12:10.491103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:10.491068    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b" event={"ID":"785248a9-5836-4d30-a522-0be624cf1b3f","Type":"ContainerStarted","Data":"f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6"}
Nov 02 14:12:10.491297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:10.491132    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b" event={"ID":"785248a9-5836-4d30-a522-0be624cf1b3f","Type":"ContainerStarted","Data":"2aa077f5ee09613d5049993ede417d2325d690590c3c9e3acfbe1b06338594cc"}
Nov 02 14:12:10.491297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:10.491199    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:11.134732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:11.134678    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:11.374435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:11.374392    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:13.013435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:13.013405    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:14.808943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:14.808912    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:16.503326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:16.503299    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:12:16.517674 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:12:16.517632    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:12:16.559575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:16.559514    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b" podStartSLOduration=8.225197649 podStartE2EDuration="8.559492782s" totalImagesPullingTime="334.295133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:10.56073511 +0000 UTC m=+895.104450705" watchObservedRunningTime="2026-11-02 14:12:16.559492782 +0000 UTC m=+901.103208377"
Nov 02 14:12:16.566322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:16.566288    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:16.709629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:16.709587    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:18.443232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:18.443201    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:20.641213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:20.641178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:22.442453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.442422    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:22.865818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.865737    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"]
Nov 02 14:12:22.871082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.871059    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:22.901801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.901766    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"]
Nov 02 14:12:22.907313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.907273    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"]
Nov 02 14:12:22.912843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.912816    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:22.931879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:22.931854    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"]
Nov 02 14:12:23.017322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017284    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.017322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017323    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtht\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.017625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017343    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017382    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017530    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.017625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017547    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017585    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017601    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017686    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017746    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.017935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017771    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.017935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017790    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.017935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017889    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.018132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017965    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.018132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.017997    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkdq\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.018132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.018029    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.018132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.018086    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.018293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.018130    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.119807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.119713    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.119807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.119755    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.119893    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.119946    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.119986    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120008    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120050    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120087    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120116    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkdq\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120154    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120157    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120226    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120303    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120521    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120537    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120676    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120689    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.120844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120824    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120882    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120914    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtht\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.120962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120941    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.121087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.120992    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.121087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.121078    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.121207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.121188    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.121315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.121219    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.121315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.121303    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.121455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.121310    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.122223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.122196    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.122861 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.122840    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.123089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.123065    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.123409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.123386    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.123532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.123514    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.142671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.142626    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.144973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.144947    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkdq\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.146012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.145991    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.146514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.146489    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtht\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.148859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.148836    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"]
Nov 02 14:12:23.154962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.154946    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.173538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.173469    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"]
Nov 02 14:12:23.177415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.177392    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4be645-openshift-default-dockercfg-2ds72\""
Nov 02 14:12:23.184621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.184599    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:23.239916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.239624    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:23.325175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.325019    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.325558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.325487    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.325800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.325700    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.325800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.325772    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.326208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.326096    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.326830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.326444    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.326830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.326502    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4hx\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.326830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.326532    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.326830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.326561    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.427839 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.427810    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428017 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.427870    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428017 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.427985    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428022    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4hx\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428050    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428080    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428138    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428215    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428259    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428333    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428428    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428608    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428619    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.428765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.428744    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.430641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.430621    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.430788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.430766    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.449691 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.449659    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.450029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.449707    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4hx\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.479655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.479626    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:23.530380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.530137    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"]
Nov 02 14:12:23.537970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.537912    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.538133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.538024    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.538133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.538074    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.586512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.586485    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"]
Nov 02 14:12:23.593119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.593074    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.593237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.593179    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.593295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.593231    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.715122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.715090    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" event={"ID":"2aa13a40-bd00-490f-9c8a-a7a915a7198c","Type":"ContainerStarted","Data":"8b1cc1b9fe51a17827b759e83e20ee3b3bc8b9f310243340481a243b068cfd0a"}
Nov 02 14:12:23.715280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.715152    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" event={"ID":"2aa13a40-bd00-490f-9c8a-a7a915a7198c","Type":"ContainerStarted","Data":"9cc110fde97d7a933cc53aa1bf9ef78a589380eb11ec3f53f4c754c6fca98426"}
Nov 02 14:12:23.718512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.718478    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" event={"ID":"a3d1901a-51b6-4350-9c3d-4c4827d10296","Type":"ContainerStarted","Data":"f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28"}
Nov 02 14:12:23.718649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.718580    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" event={"ID":"a3d1901a-51b6-4350-9c3d-4c4827d10296","Type":"ContainerStarted","Data":"dec679de82cf4c845b8f84eb191f94e9baa58163ed1d1423422bf61662f247e6"}
Nov 02 14:12:23.767590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.767390    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podStartSLOduration=1.767294858 podStartE2EDuration="1.767294858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:23.767196044 +0000 UTC m=+908.310911638" watchObservedRunningTime="2026-11-02 14:12:23.767294858 +0000 UTC m=+908.311010453"
Nov 02 14:12:23.793635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.793443    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"]
Nov 02 14:12:23.796822 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:12:23.796785    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7ca76f_2ea1_491b_bd7c_58b330e35ab9.slice/crio-f443b2f8c662410385449ffd1d3e5010255b0b85524bd7ada6e256b0b48a0dc0 WatchSource:0}: Error finding container f443b2f8c662410385449ffd1d3e5010255b0b85524bd7ada6e256b0b48a0dc0: Status 404 returned error can't find the container with id f443b2f8c662410385449ffd1d3e5010255b0b85524bd7ada6e256b0b48a0dc0
Nov 02 14:12:23.800184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.800146    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.800297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.800232    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.800297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.800279    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:23.814395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:23.814309    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podStartSLOduration=1.814289092 podStartE2EDuration="1.814289092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:23.808308725 +0000 UTC m=+908.352024331" watchObservedRunningTime="2026-11-02 14:12:23.814289092 +0000 UTC m=+908.358004678"
Nov 02 14:12:24.185233 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.185104    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:24.186453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.186426    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:24.186568 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.186485    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:24.242741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.242710    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:24.265532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.265494    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:24.265772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.265743    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:24.337992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.337959    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:24.731611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.731583    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" event={"ID":"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9","Type":"ContainerStarted","Data":"ccef464c7f47addc54e9a87d0daa5a83902a760310927eee7f281d4e3f0c5528"}
Nov 02 14:12:24.732046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.732028    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" event={"ID":"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9","Type":"ContainerStarted","Data":"f443b2f8c662410385449ffd1d3e5010255b0b85524bd7ada6e256b0b48a0dc0"}
Nov 02 14:12:24.826572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:24.824929    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podStartSLOduration=1.8248962789999998 podStartE2EDuration="1.824896279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:24.822963132 +0000 UTC m=+909.366678759" watchObservedRunningTime="2026-11-02 14:12:24.824896279 +0000 UTC m=+909.368611876"
Nov 02 14:12:25.188533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.188500    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:25.188822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.188790    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:25.245490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.245433    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:25.245805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.245762    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:25.485586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.485424    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:25.485945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.485912    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 14:12:25.486148 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:25.486107    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 14:12:26.225201 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.225157    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:26.225772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.225230    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:26.244619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.244580    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:26.244887 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.244835    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:26.529752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.529655    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 14:12:26.530022 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.529991    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 14:12:26.574747 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:26.574717    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:27.196254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:27.195777    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:27.196254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:27.195847    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:27.246012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:27.245875    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:27.246012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:27.245949    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:27.489130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:27.489040    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 14:12:27.494496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:27.494447    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 14:12:28.186914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.186876    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:28.187228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.187197    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:28.242383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.240822    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:28.242383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.240899    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:28.493439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.482471    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 14:12:28.493439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.482563    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 14:12:28.718390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:28.717327    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:29.210380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:29.209768    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:29.210380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:29.209850    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:29.245388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:29.240767    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:29.245388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:29.240831    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:29.523658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:29.523553    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 14:12:29.524247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:29.524212    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 14:12:30.211602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.211554    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:30.211822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.211643    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:30.243909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.243780    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:30.243909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.243847    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:30.291693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.291632    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:30.480892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.480791    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 14:12:30.481184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.481154    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 14:12:30.763949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:30.763860    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:31.185969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.185930    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:12:31.186190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.186000    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:12:31.245408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.240999    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:12:31.245408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.241058    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:12:31.492435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.491791    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:31.492603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.492558    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:31.493482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.493465    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:12:31.524876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.524816    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:31.643564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.643461    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:31.817694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:31.817600    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:32.162488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:32.162406    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:32.192883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:32.192843    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:32.193745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:32.193719    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:32.194569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:32.194548    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:12:33.163735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:33.163617    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:33.300217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:33.299508    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:12:33.300217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:33.299574    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:12:33.338153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:33.338077    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:33.495400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:33.488115    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:34.352378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:34.346566    4492 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:12:34.352378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:34.346637    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:12:35.037581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:35.037544    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:35.038546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:35.038430    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:35.039693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:35.039671    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:12:35.142731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:35.142693    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:35.861947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:35.861907    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:37.646596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:37.646565    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:38.083936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.083900    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"]
Nov 02 14:12:38.088372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.088340    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.119530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.119495    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"]
Nov 02 14:12:38.150154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150114    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150165    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150196    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150230    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcx4n\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150292    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150409    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150439    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150544    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.150687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.150588    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.203986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.203945    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"]
Nov 02 14:12:38.208156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.208135    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.230156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.230124    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"]
Nov 02 14:12:38.251639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.251593    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.251836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.251677    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.251836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.251711    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.251836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.251818    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.251865    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.251973    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.252001    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.252025    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.252066    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcx4n\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.252534    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.252746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.252716    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.253022 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.252997    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.253579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.253556    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.256152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.256107    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.259517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.256407    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.259517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.257654    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.275715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.275685    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.275861 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.275787    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcx4n\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.353389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353256    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353321    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdz8\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353347    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353663 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353495    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353663 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353546    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353681    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353811    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353900 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353885    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.353958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.353930    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.402457 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.402421    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:38.455670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455640    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdz8\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.455880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455689    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.455880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455736    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.455880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455769    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.455880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455829    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.456090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455918    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.456090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.455991    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.456090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.456019    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.456244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.456136    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.456585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.456561    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.457090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.457064    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.457393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.457372    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.457621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.457602    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.458102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.458078    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.460324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.460300    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.461201 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.461183    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.507245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.506605    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.507245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.507159    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdz8\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.522313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.522164    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:38.739522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.739493    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"]
Nov 02 14:12:38.740270 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:12:38.740245    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d7fbd3_b8d1_42d7_b970_508195d497bd.slice/crio-c2c53af22c13bb194dd56969f180d0d39b0d052bc2764f8bff4f0adb7fbd542c WatchSource:0}: Error finding container c2c53af22c13bb194dd56969f180d0d39b0d052bc2764f8bff4f0adb7fbd542c: Status 404 returned error can't find the container with id c2c53af22c13bb194dd56969f180d0d39b0d052bc2764f8bff4f0adb7fbd542c
Nov 02 14:12:38.771150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.771094    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:38.771308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.771246    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:38.771426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.771335    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:38.852730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.852700    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"]
Nov 02 14:12:38.853214 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:12:38.853176    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee95c24_51dc_4764_b4bf_7e1eae8a0e4a.slice/crio-7665c0bad7c58532fda9fb7b858aad82316379d3d107fde7c5c0ffc231caa893 WatchSource:0}: Error finding container 7665c0bad7c58532fda9fb7b858aad82316379d3d107fde7c5c0ffc231caa893: Status 404 returned error can't find the container with id 7665c0bad7c58532fda9fb7b858aad82316379d3d107fde7c5c0ffc231caa893
Nov 02 14:12:38.856780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.856738    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:38.856883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.856825    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:38.856883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:38.856869    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:12:39.470679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.470637    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" event={"ID":"40d7fbd3-b8d1-42d7-b970-508195d497bd","Type":"ContainerStarted","Data":"d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501"}
Nov 02 14:12:39.470869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.470701    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" event={"ID":"40d7fbd3-b8d1-42d7-b970-508195d497bd","Type":"ContainerStarted","Data":"c2c53af22c13bb194dd56969f180d0d39b0d052bc2764f8bff4f0adb7fbd542c"}
Nov 02 14:12:39.481746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.481714    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" event={"ID":"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a","Type":"ContainerStarted","Data":"67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6"}
Nov 02 14:12:39.482030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.482002    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" event={"ID":"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a","Type":"ContainerStarted","Data":"7665c0bad7c58532fda9fb7b858aad82316379d3d107fde7c5c0ffc231caa893"}
Nov 02 14:12:39.527929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.525192    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:39.527929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.527567    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:39.527929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.527623    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:39.529139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.529078    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podStartSLOduration=1.529058155 podStartE2EDuration="1.529058155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:39.5246584 +0000 UTC m=+924.068373995" watchObservedRunningTime="2026-11-02 14:12:39.529058155 +0000 UTC m=+924.072773750"
Nov 02 14:12:39.549675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.547462    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:39.594238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:39.594162    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podStartSLOduration=1.594138338 podStartE2EDuration="1.594138338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:39.573947625 +0000 UTC m=+924.117663221" watchObservedRunningTime="2026-11-02 14:12:39.594138338 +0000 UTC m=+924.137853936"
Nov 02 14:12:40.402669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:40.402634    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:40.404110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:40.404080    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:40.404215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:40.404142    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:40.522721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:40.522689    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:40.522882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:40.522737    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:41.405381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:41.404875    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:41.405381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:41.404956    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:41.523302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:41.523263    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:41.523477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:41.523330    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:41.615955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:41.615923    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:42.406952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:42.406915    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:42.407541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:42.407511    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:42.527956 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:42.527918    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:42.528239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:42.528205    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:43.405415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:43.405134    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:43.405415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:43.405195    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:43.527396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:43.524035    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:43.527396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:43.524112    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:44.303085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:44.303036    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:44.406946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:44.406913    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:44.407234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:44.407184    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:44.529711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:44.529678    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:44.530315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:44.530268    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:45.402812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:45.402776    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:45.402993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:45.402835    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:45.522696 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:45.522656    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:45.522870 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:45.522708    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:46.235060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:46.235024    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:46.403447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:46.403412    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:12:46.403741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:46.403707    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:12:46.523277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:46.523170    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 14:12:46.523916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:46.523250    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 14:12:46.565080 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:12:46.565048    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:12:47.407710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:47.407678    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:47.408143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:47.407954    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:47.408841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:47.408823    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:12:47.529471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:47.529436    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:47.530551 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:47.530529    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:47.531343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:47.531323    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:12:48.084712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:48.084674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:50.047466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:50.047433    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:51.824735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:51.824679    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:53.626476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:53.626445    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:55.144833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:55.144805    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:55.405725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:55.405639    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:56.008945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:56.008895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:56.169438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:56.166147    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:12:58.185310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:12:58.185266    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:00.025654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:00.025611    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:01.035647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:01.035607    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:01.166903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:01.166851    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:02.026599 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.026540    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:02.806231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.806166    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:02.931092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.931064    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:02.970109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.970079    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"]
Nov 02 14:13:02.974519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.974496    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:02.995307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.995271    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"]
Nov 02 14:13:02.999888 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:02.999864    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-de8cc7\""
Nov 02 14:13:03.000189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.000174    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-de8cc7-dockercfg-64pbb\""
Nov 02 14:13:03.021626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.021592    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfqs\" (UniqueName: \"kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs\") pod \"payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.023247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.023216    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.138787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.124372    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.142110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.142083    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfqs\" (UniqueName: \"kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs\") pod \"payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.159642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.141927    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.160116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.139287    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"]
Nov 02 14:13:03.212059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.210343    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.214500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.214470    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfqs\" (UniqueName: \"kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs\") pod \"payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.260621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.260587    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sb6k\" (UniqueName: \"kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k\") pod \"payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.261068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.261040    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.278662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.278625    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"]
Nov 02 14:13:03.302592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.302544    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:03.362681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.362642    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sb6k\" (UniqueName: \"kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k\") pod \"payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.362839 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.362811    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.363592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.363568    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.402818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.402745    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sb6k\" (UniqueName: \"kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k\") pod \"payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.605475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.605440    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:03.634191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.634145    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"]
Nov 02 14:13:03.925062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.925013    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:03.936385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:03.936313    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"]
Nov 02 14:13:03.939014 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:13:03.938980    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59765dea_71c1_448c_bc17_cbde7df8b505.slice/crio-f8e5de303ed75c2f5d38bea92d7ea3f15e29a6590e8e77b20f88d43dec75c12e WatchSource:0}: Error finding container f8e5de303ed75c2f5d38bea92d7ea3f15e29a6590e8e77b20f88d43dec75c12e: Status 404 returned error can't find the container with id f8e5de303ed75c2f5d38bea92d7ea3f15e29a6590e8e77b20f88d43dec75c12e
Nov 02 14:13:04.216423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.216308    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" event={"ID":"c76db8eb-56c5-4d19-a55b-63dedde658c7","Type":"ContainerStarted","Data":"f88ec4ecd7de1b8e62de5864c5a9c0e6e42ce18f0f14f6d4aec8774a09ee09ef"}
Nov 02 14:13:04.216423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.216391    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" event={"ID":"c76db8eb-56c5-4d19-a55b-63dedde658c7","Type":"ContainerStarted","Data":"bb53b0e56139844beb57f45986c82d0fafb782c0eeb418c68b5154060f833076"}
Nov 02 14:13:04.216787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.216764    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:04.218669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.218629    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt" event={"ID":"59765dea-71c1-448c-bc17-cbde7df8b505","Type":"ContainerStarted","Data":"5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd"}
Nov 02 14:13:04.218801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.218692    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt" event={"ID":"59765dea-71c1-448c-bc17-cbde7df8b505","Type":"ContainerStarted","Data":"f8e5de303ed75c2f5d38bea92d7ea3f15e29a6590e8e77b20f88d43dec75c12e"}
Nov 02 14:13:04.218801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.218721    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:04.256914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.256851    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" podStartSLOduration=2.25683149 podStartE2EDuration="2.25683149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:04.252848572 +0000 UTC m=+948.796564166" watchObservedRunningTime="2026-11-02 14:13:04.25683149 +0000 UTC m=+948.800547086"
Nov 02 14:13:04.289671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:04.289572    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt" podStartSLOduration=1.289555509 podStartE2EDuration="1.289555509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:04.28938455 +0000 UTC m=+948.833100145" watchObservedRunningTime="2026-11-02 14:13:04.289555509 +0000 UTC m=+948.833271104"
Nov 02 14:13:05.744161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:05.744121    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:05.901643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:05.901609    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:06.584576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:06.584542    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:07.576235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:07.576204    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:09.425549 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:09.425512    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:11.123800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:11.123766    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:11.453328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:11.453296    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:12.824089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:12.824057    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:14.588681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:14.588644    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:14.721202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:14.721172    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:15.040866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:15.040833    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:15.245595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:15.245559    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:13:15.247189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:15.247112    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:13:16.556132 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:13:16.556090    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:13:16.674170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:16.674130    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:16.790339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:16.790300    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:18.460554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:18.460522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:19.171444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:19.171405    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:19.754538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:19.754508    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:20.420450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:20.420413    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:22.245215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:22.245178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:23.364145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.364111    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"]
Nov 02 14:13:23.371041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.371015    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.407270 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.400896    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"]
Nov 02 14:13:23.453513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453478    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.453705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453524    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.453705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453602    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.453784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453753    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.453817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453798    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.453859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453831    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn594\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.453968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.453945    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.454049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.454035    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.454154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.454140    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.554877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.554846    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.554896    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.554921    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn594\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.554961    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555009    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555279 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555089    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555279 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555146    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555279 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555171    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555279 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555205    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555551 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555527    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555761    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555787    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.555886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.555764    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.556335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.556289    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.558002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.557983    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.558078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.558014    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.577553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.577526    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.577659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.577551    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn594\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:23.687840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:23.687804    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:24.018147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.018117    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"]
Nov 02 14:13:24.020123 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:13:24.020064    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3d5ac5_3b5d_4e4f_917c_e44c137c453d.slice/crio-d9b05cc5bebbcdf2ccfc10c06df11e74c6c894e8037d24ba9ad72f1a04727422 WatchSource:0}: Error finding container d9b05cc5bebbcdf2ccfc10c06df11e74c6c894e8037d24ba9ad72f1a04727422: Status 404 returned error can't find the container with id d9b05cc5bebbcdf2ccfc10c06df11e74c6c894e8037d24ba9ad72f1a04727422
Nov 02 14:13:24.022815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.022780    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:13:24.022904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.022853    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:13:24.022904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.022888    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:13:24.043741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.043706    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:24.441078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.441032    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" event={"ID":"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d","Type":"ContainerStarted","Data":"6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c"}
Nov 02 14:13:24.441599 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.441177    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" event={"ID":"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d","Type":"ContainerStarted","Data":"d9b05cc5bebbcdf2ccfc10c06df11e74c6c894e8037d24ba9ad72f1a04727422"}
Nov 02 14:13:24.688615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:24.688570    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:25.690113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:25.690075    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:25.690651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:25.690141    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:25.772540 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:25.772464    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podStartSLOduration=2.77244077 podStartE2EDuration="2.77244077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:24.503284338 +0000 UTC m=+969.046999931" watchObservedRunningTime="2026-11-02 14:13:25.77244077 +0000 UTC m=+970.316156366"
Nov 02 14:13:25.773733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:25.773707    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"]
Nov 02 14:13:25.804046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:25.804014    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"]
Nov 02 14:13:25.922652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:25.922619    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:26.055432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:26.054932    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:26.312122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:26.312028    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:26.614477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:26.613895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:26.694122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:26.694085    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:26.694713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:26.694681    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:27.688517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:27.688476    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:27.688806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:27.688780    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:28.689379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:28.688667    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:28.689379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:28.688732    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:29.806589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:29.806552    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:29.809675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:29.809643    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:30.097562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:30.097485    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:30.688932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:30.688892    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:30.689147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:30.688956    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:30.774251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:30.774206    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerName="payload-processing" containerID="cri-o://58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204" gracePeriod=25
Nov 02 14:13:30.805733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:30.805682    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerName="payload-pre-processing" containerID="cri-o://333fd7470a14a2c62d73f4e614b88375a8b74b2a187d4911be26b7241146be94" gracePeriod=25
Nov 02 14:13:31.688344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:31.688299    4492 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 14:13:31.688741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:31.688406    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 14:13:32.599672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:32.599637    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:33.089113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:33.088828    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:33.090858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:33.090837    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:33.091642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:33.091612    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:13:33.481710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:33.481674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:34.664444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:34.664405    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:36.621384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:36.621231    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:36.754087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:36.754048    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:36.978391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:36.968199    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:37.144386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.136917    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:37.176629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.176586    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:37.901193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.901152    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"]
Nov 02 14:13:37.906688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.906666    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:37.921971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.921943    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"]
Nov 02 14:13:37.926800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.926776    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-2a51f04e5b66\""
Nov 02 14:13:37.926959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.926879    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-f650d0393004\""
Nov 02 14:13:37.927025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.926969    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-e2e--c0104787-dockercfg-tx5xz\""
Nov 02 14:13:37.927025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.926981    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bb4ef\""
Nov 02 14:13:37.927128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.927080    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"kube-root-ca.crt\""
Nov 02 14:13:37.930766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.930742    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"openshift-service-ca.crt\""
Nov 02 14:13:37.930919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:37.930798    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"routing-overlay\""
Nov 02 14:13:38.060425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.060344    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.060603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.060503    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.060683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.060660    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vfs\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.060724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.060697    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.161539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.161451    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.161689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.161574    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vfs\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.161689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.161597    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.161821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.161686    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.162246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.162218    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.162503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.162446    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.164333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.164310    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.183489 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.183460    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vfs\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.223654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.223619    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:38.537822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.537790    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"]
Nov 02 14:13:38.539156 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:13:38.539130    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ae08f3_b661_4879_a3c3_52b838de1718.slice/crio-f0385c15ae48d09e1b70bddda43dad980765da1f9479aeddb20b15007ba020a1 WatchSource:0}: Error finding container f0385c15ae48d09e1b70bddda43dad980765da1f9479aeddb20b15007ba020a1: Status 404 returned error can't find the container with id f0385c15ae48d09e1b70bddda43dad980765da1f9479aeddb20b15007ba020a1
Nov 02 14:13:38.738639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:38.738603    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:39.014967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:39.014917    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8" event={"ID":"93ae08f3-b661-4879-a3c3-52b838de1718","Type":"ContainerStarted","Data":"dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434"}
Nov 02 14:13:39.015559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:39.014986    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8" event={"ID":"93ae08f3-b661-4879-a3c3-52b838de1718","Type":"ContainerStarted","Data":"f0385c15ae48d09e1b70bddda43dad980765da1f9479aeddb20b15007ba020a1"}
Nov 02 14:13:39.016018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:39.015995    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:39.053933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:39.053872    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8" podStartSLOduration=2.053857284 podStartE2EDuration="2.053857284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:39.052731323 +0000 UTC m=+983.596446919" watchObservedRunningTime="2026-11-02 14:13:39.053857284 +0000 UTC m=+983.597572878"
Nov 02 14:13:39.266935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:39.266839    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:40.782416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:40.782379    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"]
Nov 02 14:13:40.782819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:40.782592    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" podUID="ac1a136a-3003-467b-8fb6-70e051b39d73" containerName="maas-api" containerID="cri-o://a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8" gracePeriod=30
Nov 02 14:13:40.857949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:40.857918    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:41.547100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.547070    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:13:41.631702 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.631665    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\") pod \"ac1a136a-3003-467b-8fb6-70e051b39d73\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") "
Nov 02 14:13:41.631891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.631826    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg\" (UniqueName: \"kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg\") pod \"ac1a136a-3003-467b-8fb6-70e051b39d73\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") "
Nov 02 14:13:41.631955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.631930    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca\") pod \"ac1a136a-3003-467b-8fb6-70e051b39d73\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") "
Nov 02 14:13:41.632018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.631984    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs\") pod \"ac1a136a-3003-467b-8fb6-70e051b39d73\" (UID: \"ac1a136a-3003-467b-8fb6-70e051b39d73\") "
Nov 02 14:13:41.632912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.632390    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca" pod "ac1a136a-3003-467b-8fb6-70e051b39d73" (UID: "ac1a136a-3003-467b-8fb6-70e051b39d73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:41.632912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.632828    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac1a136a-3003-467b-8fb6-70e051b39d73-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:41.634791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.634751    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs" pod "ac1a136a-3003-467b-8fb6-70e051b39d73" (UID: "ac1a136a-3003-467b-8fb6-70e051b39d73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:41.636093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.636072    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls" pod "ac1a136a-3003-467b-8fb6-70e051b39d73" (UID: "ac1a136a-3003-467b-8fb6-70e051b39d73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:41.651584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.651559    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg" pod "ac1a136a-3003-467b-8fb6-70e051b39d73" (UID: "ac1a136a-3003-467b-8fb6-70e051b39d73"). InnerVolumeSpecName "kube-api-access-s6xfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:41.666348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.666306    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:41.747478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.745698    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:41.747478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.745735    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac1a136a-3003-467b-8fb6-70e051b39d73-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:41.747478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.745751    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xfg\" (UniqueName: \"kubernetes.io/projected/ac1a136a-3003-467b-8fb6-70e051b39d73-kube-api-access-s6xfg\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:41.798052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:41.798018    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:42.038581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.038501    4492 generic.go:356] "Generic (PLEG): container finished" podID="ac1a136a-3003-467b-8fb6-70e051b39d73" containerID="a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8" exitCode=0
Nov 02 14:13:42.038581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.038579    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" event={"ID":"ac1a136a-3003-467b-8fb6-70e051b39d73","Type":"ContainerDied","Data":"a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8"}
Nov 02 14:13:42.038770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.038646    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw" event={"ID":"ac1a136a-3003-467b-8fb6-70e051b39d73","Type":"ContainerDied","Data":"db0f160aadb4418a435f40bf0f92ed8af98f86aec889e948150f5763aae02cac"}
Nov 02 14:13:42.038770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.038642    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"
Nov 02 14:13:42.038770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.038668    4492 scope.go:122] "RemoveContainer" containerID="a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8"
Nov 02 14:13:42.050196 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.050178    4492 scope.go:122] "RemoveContainer" containerID="a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8"
Nov 02 14:13:42.050512 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:13:42.050489    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8\": container with ID starting with a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8 not found: ID does not exist" containerID="a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8"
Nov 02 14:13:42.050618 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.050519    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8"} err="failed to get container status \"a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8\": rpc error: code = NotFound desc = could not find container \"a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8\": container with ID starting with a224405def19b5416e82813146f6a281540a5d425b92064ce25d75fb311086b8 not found: ID does not exist"
Nov 02 14:13:42.091579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.091544    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"]
Nov 02 14:13:42.100845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.100818    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-758617-75bdccd74f-v8fxw"]
Nov 02 14:13:42.983255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:42.983215    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:43.431969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.431939    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"]
Nov 02 14:13:43.436397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.432829    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ac1a136a-3003-467b-8fb6-70e051b39d73" containerName="maas-api"
Nov 02 14:13:43.436397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.432853    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1a136a-3003-467b-8fb6-70e051b39d73" containerName="maas-api"
Nov 02 14:13:43.436397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.433020    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1a136a-3003-467b-8fb6-70e051b39d73" containerName="maas-api"
Nov 02 14:13:43.438898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.438876    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.460989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.460965    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-514524-openshift-default-dockercfg-5bz72\""
Nov 02 14:13:43.468663 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.468633    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"]
Nov 02 14:13:43.503658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.503621    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:43.590499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.574308    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7k6w\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.590912    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591006    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591046    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591089    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591136    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591217    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591242    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.626507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.591416    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.643930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.643901    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:43.666880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.666836    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1a136a-3003-467b-8fb6-70e051b39d73" path="/var/lib/kubelet/pods/ac1a136a-3003-467b-8fb6-70e051b39d73/volumes"
Nov 02 14:13:43.697572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.697471    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.697890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.697868    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7k6w\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698041    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698178    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698287    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698419    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698541    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698680    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.698781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.698769    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.699750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.699724    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.700168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.700148    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.717480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.703719    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.717713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.705168    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.717814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.705390    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.727715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.723464    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.735728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.735661    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7k6w\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.736456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.736409    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.746332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.746264    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo\") pod \"e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") " pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.759168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.758895    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:43.800381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:43.794890    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:44.272797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:44.272770    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"]
Nov 02 14:13:44.275185 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:13:44.275154    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dcb0fb_68a0_4605_b953_5426cb18014a.slice/crio-b064d051738dcd7fbd38ec8ddd27f0356119b5133ff5f4ba120d4c91cc760cbc WatchSource:0}: Error finding container b064d051738dcd7fbd38ec8ddd27f0356119b5133ff5f4ba120d4c91cc760cbc: Status 404 returned error can't find the container with id b064d051738dcd7fbd38ec8ddd27f0356119b5133ff5f4ba120d4c91cc760cbc
Nov 02 14:13:44.278030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:44.277994    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:13:44.278106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:44.278065    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:13:44.278106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:44.278095    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:13:45.115381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:45.110303    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" event={"ID":"00dcb0fb-68a0-4605-b953-5426cb18014a","Type":"ContainerStarted","Data":"87581e28fba08cd11cdff6a2c24e1d26b44448e651bc4709c998c79e1a0b2e7f"}
Nov 02 14:13:45.115381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:45.110402    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" event={"ID":"00dcb0fb-68a0-4605-b953-5426cb18014a","Type":"ContainerStarted","Data":"b064d051738dcd7fbd38ec8ddd27f0356119b5133ff5f4ba120d4c91cc760cbc"}
Nov 02 14:13:45.182379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:45.181330    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podStartSLOduration=2.181310826 podStartE2EDuration="2.181310826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:45.169179971 +0000 UTC m=+989.712895562" watchObservedRunningTime="2026-11-02 14:13:45.181310826 +0000 UTC m=+989.725026422"
Nov 02 14:13:45.760545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:45.760494    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:45.767546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:45.767515    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:45.796005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:45.795967    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:46.120413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:46.120298    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:46.121645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:46.121621    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:13:46.624095 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:13:46.624053    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:13:47.096217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:47.096177    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:47.096675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:47.096248    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:47.845854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:47.845811    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:49.645197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:49.645165    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:49.990926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:49.990887    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:50.118405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:50.118336    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:50.818435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:50.818403    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:51.035781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.034562    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:13:51.138563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.133524    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:13:51.164839 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.164799    4492 generic.go:356] "Generic (PLEG): container finished" podID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerID="58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204" exitCode=0
Nov 02 14:13:51.164985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.164926    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" event={"ID":"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48","Type":"ContainerDied","Data":"58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204"}
Nov 02 14:13:51.165049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.164984    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h" event={"ID":"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48","Type":"ContainerDied","Data":"aa4c49ec89cb8ba0d52e4f8bd8ab5cfb7b4491f6d6721b4609c061408079fca7"}
Nov 02 14:13:51.165049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.165013    4492 scope.go:122] "RemoveContainer" containerID="58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204"
Nov 02 14:13:51.165501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.165448    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"
Nov 02 14:13:51.167297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.167274    4492 generic.go:356] "Generic (PLEG): container finished" podID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerID="333fd7470a14a2c62d73f4e614b88375a8b74b2a187d4911be26b7241146be94" exitCode=0
Nov 02 14:13:51.167473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.167456    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" event={"ID":"f86f680c-2f36-47fa-811b-041a12a46fd4","Type":"ContainerDied","Data":"333fd7470a14a2c62d73f4e614b88375a8b74b2a187d4911be26b7241146be94"}
Nov 02 14:13:51.178293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.178265    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume\") pod \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") "
Nov 02 14:13:51.178453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.178437    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml\" (UniqueName: \"kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml\") pod \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\" (UID: \"f2ebe684-4231-4d1c-8896-f3e5ab3dbe48\") "
Nov 02 14:13:51.179127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.179055    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume" pod "f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" (UID: "f2ebe684-4231-4d1c-8896-f3e5ab3dbe48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:51.181435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.181411    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml" pod "f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" (UID: "f2ebe684-4231-4d1c-8896-f3e5ab3dbe48"). InnerVolumeSpecName "kube-api-access-j8bml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:51.188737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.187379    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:13:51.191091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.191073    4492 scope.go:122] "RemoveContainer" containerID="58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204"
Nov 02 14:13:51.191467 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:13:51.191447    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204\": container with ID starting with 58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204 not found: ID does not exist" containerID="58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204"
Nov 02 14:13:51.191536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.191476    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204"} err="failed to get container status \"58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204\": rpc error: code = NotFound desc = could not find container \"58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204\": container with ID starting with 58be3325583a259d8edd6cbcfe06079066ad2d9e9a030dd0a5e93ac100b13204 not found: ID does not exist"
Nov 02 14:13:51.280302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.280268    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m\") pod \"f86f680c-2f36-47fa-811b-041a12a46fd4\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") "
Nov 02 14:13:51.280302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.280309    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume\") pod \"f86f680c-2f36-47fa-811b-041a12a46fd4\" (UID: \"f86f680c-2f36-47fa-811b-041a12a46fd4\") "
Nov 02 14:13:51.280982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.280949    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.280982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.280977    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bml\" (UniqueName: \"kubernetes.io/projected/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48-kube-api-access-j8bml\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.281193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.281068    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume" pod "f86f680c-2f36-47fa-811b-041a12a46fd4" (UID: "f86f680c-2f36-47fa-811b-041a12a46fd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:51.283124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.283097    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m" pod "f86f680c-2f36-47fa-811b-041a12a46fd4" (UID: "f86f680c-2f36-47fa-811b-041a12a46fd4"). InnerVolumeSpecName "kube-api-access-lrr6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:51.304706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.304615    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"]
Nov 02 14:13:51.304921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.304891    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" containerID="cri-o://9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24" gracePeriod=30
Nov 02 14:13:51.329443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.329404    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:51.427444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.414454    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/f86f680c-2f36-47fa-811b-041a12a46fd4-kube-api-access-lrr6m\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.427444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.414499    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f86f680c-2f36-47fa-811b-041a12a46fd4-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.611966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.611834    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"]
Nov 02 14:13:51.619185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.619156    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-758617-84bfb45777-wtq4h"]
Nov 02 14:13:51.666247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:51.666214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:52.176419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:52.176391    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"
Nov 02 14:13:52.176931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:52.176390    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7" event={"ID":"f86f680c-2f36-47fa-811b-041a12a46fd4","Type":"ContainerDied","Data":"bdaf271d01286c3e2d613ed851149bcb60e30951091a88f074746039fca2789c"}
Nov 02 14:13:52.176931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:52.176523    4492 scope.go:122] "RemoveContainer" containerID="333fd7470a14a2c62d73f4e614b88375a8b74b2a187d4911be26b7241146be94"
Nov 02 14:13:52.232636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:52.232603    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"]
Nov 02 14:13:52.241822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:52.241797    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-758617-786b6d75ff-r4sx7"]
Nov 02 14:13:53.521777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:53.521744    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:53.616480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:53.614551    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" path="/var/lib/kubelet/pods/f2ebe684-4231-4d1c-8896-f3e5ab3dbe48/volumes"
Nov 02 14:13:53.616480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:53.615172    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" path="/var/lib/kubelet/pods/f86f680c-2f36-47fa-811b-041a12a46fd4/volumes"
Nov 02 14:13:54.565585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:54.565549    4492 patch_prober.go:29] interesting pod/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:13:54.565973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:54.565648    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:13:55.331769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:55.331739    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:56.595418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.595391    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:13:56.696662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.696605    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.696873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.696686    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.696873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.696738    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.696873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.696808    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.697038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.696906    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.697038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.696948    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.697038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697026    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.697143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697058    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.697143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697017    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:56.697233 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697197    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs\") pod \"dcec7d06-a5b0-4779-a782-c714554a468d\" (UID: \"dcec7d06-a5b0-4779-a782-c714554a468d\") "
Nov 02 14:13:56.697431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697381    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:56.697649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697625    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:56.698002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697822    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:56.698002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697886    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.698002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697905    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.698002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.697929    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.698265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.698150    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:56.700475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.700448    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:56.700706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.700674    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "kube-api-access-rw9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:56.700908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.700880    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:56.703091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.703062    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo" pod "dcec7d06-a5b0-4779-a782-c714554a468d" (UID: "dcec7d06-a5b0-4779-a782-c714554a468d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:13:56.798818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.798788    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcec7d06-a5b0-4779-a782-c714554a468d-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.798818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.798817    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw9xx\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-kube-api-access-rw9xx\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.798818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.798829    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.799076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.798842    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcec7d06-a5b0-4779-a782-c714554a468d-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.799076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.798855    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcec7d06-a5b0-4779-a782-c714554a468d-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.799076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.798865    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcec7d06-a5b0-4779-a782-c714554a468d-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:13:56.879293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:56.879222    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:57.146473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.143588    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:57.216896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.216866    4492 generic.go:356] "Generic (PLEG): container finished" podID="dcec7d06-a5b0-4779-a782-c714554a468d" containerID="9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24" exitCode=0
Nov 02 14:13:57.217071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.216937    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"
Nov 02 14:13:57.217071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.216948    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" event={"ID":"dcec7d06-a5b0-4779-a782-c714554a468d","Type":"ContainerDied","Data":"9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24"}
Nov 02 14:13:57.217071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.217015    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88" event={"ID":"dcec7d06-a5b0-4779-a782-c714554a468d","Type":"ContainerDied","Data":"cf293cca3df98c706874446e0b659dfad51bbdf7091f50b0698793b7722b900f"}
Nov 02 14:13:57.217071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.217039    4492 scope.go:122] "RemoveContainer" containerID="9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24"
Nov 02 14:13:57.229128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.229110    4492 scope.go:122] "RemoveContainer" containerID="515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"
Nov 02 14:13:57.239516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.239487    4492 scope.go:122] "RemoveContainer" containerID="9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24"
Nov 02 14:13:57.239813 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:13:57.239792    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24\": container with ID starting with 9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24 not found: ID does not exist" containerID="9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24"
Nov 02 14:13:57.239901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.239825    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24"} err="failed to get container status \"9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24\": rpc error: code = NotFound desc = could not find container \"9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24\": container with ID starting with 9ecc557a3061129ff1b73de7f7f93c4b1f9b4dc6a9f52cfbaddd39e35a2c3b24 not found: ID does not exist"
Nov 02 14:13:57.239901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.239850    4492 scope.go:122] "RemoveContainer" containerID="515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"
Nov 02 14:13:57.240167 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:13:57.240146    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c\": container with ID starting with 515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c not found: ID does not exist" containerID="515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"
Nov 02 14:13:57.240213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.240175    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c"} err="failed to get container status \"515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c\": rpc error: code = NotFound desc = could not find container \"515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c\": container with ID starting with 515df6858bdc18582ab44c74ad9ff8ecc44bfa8bb56431de7329725e7638b45c not found: ID does not exist"
Nov 02 14:13:57.271889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.271849    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"]
Nov 02 14:13:57.275234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.275211    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-758617-openshift-default-754657d5bc-vcf88"]
Nov 02 14:13:57.582543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.582511    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" path="/var/lib/kubelet/pods/dcec7d06-a5b0-4779-a782-c714554a468d/volumes"
Nov 02 14:13:57.692516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.692487    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:57.812106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:57.812064    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:59.370556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.370487    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:59.557484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.554754    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:59.677692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.677657    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:13:59.699652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.699610    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"]
Nov 02 14:13:59.700288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700272    4492 cpu_manager.go:414] "containerMap: removing container" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700291    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700301    4492 cpu_manager.go:414] "containerMap: removing container" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerName="payload-pre-processing"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700308    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerName="payload-pre-processing"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700330    4492 cpu_manager.go:414] "containerMap: removing container" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerName="payload-processing"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700335    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerName="payload-processing"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700341    4492 cpu_manager.go:414] "containerMap: removing container" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy"
Nov 02 14:13:59.700349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700346    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy"
Nov 02 14:13:59.700598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700455    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ebe684-4231-4d1c-8896-f3e5ab3dbe48" containerName="payload-processing"
Nov 02 14:13:59.700598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700466    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy"
Nov 02 14:13:59.700598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.700477    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86f680c-2f36-47fa-811b-041a12a46fd4" containerName="payload-pre-processing"
Nov 02 14:13:59.705455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.705428    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:13:59.725258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.725226    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-514524\""
Nov 02 14:13:59.729042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.729012    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"]
Nov 02 14:13:59.753676 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.753645    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-514524-dockercfg-5c7lp\""
Nov 02 14:13:59.878380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.873279    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xj58\" (UniqueName: \"kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58\") pod \"payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:13:59.895429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.873347    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:13:59.897617 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.897589    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"]
Nov 02 14:13:59.910587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.910550    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcec7d06-a5b0-4779-a782-c714554a468d" containerName="istio-proxy"
Nov 02 14:13:59.915744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.915711    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.005435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:13:59.997957    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:00.075284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.029272    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fn79\" (UniqueName: \"kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79\") pod \"payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.075615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.075584    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xj58\" (UniqueName: \"kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58\") pod \"payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:00.076127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.076105    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:00.077283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.077260    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.079127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.074492    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"]
Nov 02 14:14:00.079127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.077113    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:00.110503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.105957    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xj58\" (UniqueName: \"kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58\") pod \"payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:00.180132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.180096    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fn79\" (UniqueName: \"kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79\") pod \"payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.180378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.180192    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.181111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.181082    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.196260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.196229    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:00.202189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.202167    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fn79\" (UniqueName: \"kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79\") pod \"payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:00.324688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.324597    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:00.469248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:00.469220    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:01.262438 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:01.262405    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7c71bb_bed4_4eb1_98df_055d2bf3fb4f.slice/crio-c5061ed27b81ade04951c9176c72546dd54df68eff0c4be03c36031be00f98f3 WatchSource:0}: Error finding container c5061ed27b81ade04951c9176c72546dd54df68eff0c4be03c36031be00f98f3: Status 404 returned error can't find the container with id c5061ed27b81ade04951c9176c72546dd54df68eff0c4be03c36031be00f98f3
Nov 02 14:14:01.263207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.263183    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"]
Nov 02 14:14:01.272124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.272100    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"]
Nov 02 14:14:01.277657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.277636    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.278622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.278602    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"]
Nov 02 14:14:01.282149 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:01.282128    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6c6604_cea8_4fd1_b4e8_f2bd40ac432c.slice/crio-4efacf1b965ce9ed8a518ade22ddead6aaa34013a0eb3e127bd1e5e2f802036b WatchSource:0}: Error finding container 4efacf1b965ce9ed8a518ade22ddead6aaa34013a0eb3e127bd1e5e2f802036b: Status 404 returned error can't find the container with id 4efacf1b965ce9ed8a518ade22ddead6aaa34013a0eb3e127bd1e5e2f802036b
Nov 02 14:14:01.291212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.291182    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-636908050766\""
Nov 02 14:14:01.294288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.293831    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"]
Nov 02 14:14:01.328365 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.328312    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph" event={"ID":"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f","Type":"ContainerStarted","Data":"c5061ed27b81ade04951c9176c72546dd54df68eff0c4be03c36031be00f98f3"}
Nov 02 14:14:01.329967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.329933    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk" event={"ID":"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c","Type":"ContainerStarted","Data":"4efacf1b965ce9ed8a518ade22ddead6aaa34013a0eb3e127bd1e5e2f802036b"}
Nov 02 14:14:01.393284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.393256    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:01.438383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.431442    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824lh\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.438383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.431560    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.438383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.431696    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.438383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.431773    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.533215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.533183    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.533942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.533255    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.533942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.533433    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824lh\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.533942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.533483    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.534135 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.533985    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.534826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.534735    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.536531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.536506    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.554863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.554830    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-824lh\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:01.709124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:01.709085    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:02.018344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.018242    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"]
Nov 02 14:14:02.020815 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:02.020790    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527c21f1_4cc3_45ef_8ec9_171d1b780acc.slice/crio-ed612614dadf61d47b0eba662743646a5531d764a0f2eabbe93529045cdeef8d WatchSource:0}: Error finding container ed612614dadf61d47b0eba662743646a5531d764a0f2eabbe93529045cdeef8d: Status 404 returned error can't find the container with id ed612614dadf61d47b0eba662743646a5531d764a0f2eabbe93529045cdeef8d
Nov 02 14:14:02.175400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.175335    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:02.339780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.339683    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58" event={"ID":"527c21f1-4cc3-45ef-8ec9-171d1b780acc","Type":"ContainerStarted","Data":"b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce"}
Nov 02 14:14:02.339780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.339768    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58" event={"ID":"527c21f1-4cc3-45ef-8ec9-171d1b780acc","Type":"ContainerStarted","Data":"ed612614dadf61d47b0eba662743646a5531d764a0f2eabbe93529045cdeef8d"}
Nov 02 14:14:02.339982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.339890    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:02.341440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.341410    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph" event={"ID":"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f","Type":"ContainerStarted","Data":"030991b4fb99870926fce2ae59e80dced6f6216e9d6d25d8a33bbbc093386b31"}
Nov 02 14:14:02.341579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.341459    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:02.343090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.343063    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk" event={"ID":"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c","Type":"ContainerStarted","Data":"c9701f0cc852485ce73ed37ee1fe50badbe278f3ea55db9e5ce0778b0c666c11"}
Nov 02 14:14:02.343193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.343146    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:02.380004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.379951    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58" podStartSLOduration=1.379935678 podStartE2EDuration="1.379935678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.376825594 +0000 UTC m=+1006.920541187" watchObservedRunningTime="2026-11-02 14:14:02.379935678 +0000 UTC m=+1006.923651269"
Nov 02 14:14:02.419639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.419589    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk" podStartSLOduration=3.419573392 podStartE2EDuration="3.419573392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.415296449 +0000 UTC m=+1006.959012054" watchObservedRunningTime="2026-11-02 14:14:02.419573392 +0000 UTC m=+1006.963289011"
Nov 02 14:14:02.452454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:02.452404    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph" podStartSLOduration=3.452388132 podStartE2EDuration="3.452388132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.451411974 +0000 UTC m=+1006.995127569" watchObservedRunningTime="2026-11-02 14:14:02.452388132 +0000 UTC m=+1006.996103725"
Nov 02 14:14:03.051471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.051428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:03.185381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.184980    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:03.298184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.298144    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:03.322634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.322558    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"]
Nov 02 14:14:03.327010 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.326987    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.376926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.370820    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"payload-processing-external-model-e2e--d319af1f-dockercfg-bm9qt\""
Nov 02 14:14:03.376926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.370930    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"routing-overlay\""
Nov 02 14:14:03.376926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.371020    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"openshift-service-ca.crt\""
Nov 02 14:14:03.376926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.371055    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"agc-cred-f1de9c615677\""
Nov 02 14:14:03.376926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.371090    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bf0b0d\""
Nov 02 14:14:03.376926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.371146    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"kube-root-ca.crt\""
Nov 02 14:14:03.391314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.384826    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"]
Nov 02 14:14:03.563382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.542333    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.627447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.624701    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.627447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.624803    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85h8\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.627447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.624924    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.726537 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.726494    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.726756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.726567    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.726756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.726633    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85h8\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.726756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.726720    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.727409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.727376    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.727558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.727537    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.729474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.729446    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.749266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.749228    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85h8\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:03.976501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:03.976459    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:04.284670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:04.284643    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"]
Nov 02 14:14:04.287420 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:04.287385    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9db4a2c_b160_44a4_b371_d1273b825aeb.slice/crio-e1c4b9efa3b26ae1c07011ce694b4d1d80062c7c4bd51aa8379544c5be627412 WatchSource:0}: Error finding container e1c4b9efa3b26ae1c07011ce694b4d1d80062c7c4bd51aa8379544c5be627412: Status 404 returned error can't find the container with id e1c4b9efa3b26ae1c07011ce694b4d1d80062c7c4bd51aa8379544c5be627412
Nov 02 14:14:04.381105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:04.381066    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw" event={"ID":"a9db4a2c-b160-44a4-b371-d1273b825aeb","Type":"ContainerStarted","Data":"3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735"}
Nov 02 14:14:04.381295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:04.381128    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw" event={"ID":"a9db4a2c-b160-44a4-b371-d1273b825aeb","Type":"ContainerStarted","Data":"e1c4b9efa3b26ae1c07011ce694b4d1d80062c7c4bd51aa8379544c5be627412"}
Nov 02 14:14:04.381295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:04.381155    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:04.422344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:04.422288    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw" podStartSLOduration=1.422267363 podStartE2EDuration="1.422267363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:04.417974914 +0000 UTC m=+1008.961690508" watchObservedRunningTime="2026-11-02 14:14:04.422267363 +0000 UTC m=+1008.965982958"
Nov 02 14:14:05.085024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:05.084988    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:06.700868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:06.700835    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:06.808145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:06.808114    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:08.345958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:08.345919    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"]
Nov 02 14:14:08.472870 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:08.472839    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:08.657434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:08.653437    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:08.767743 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:08.767711    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:09.620383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:09.615007    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:11.542844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:11.542777    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:11.665596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:11.665497    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:13.193503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.193467    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"]
Nov 02 14:14:13.197802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.197780    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.215662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.215633    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-514524-serving-cert\""
Nov 02 14:14:13.231402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.227650    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"]
Nov 02 14:14:13.249343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.249287    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.250372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.250324    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcmc\" (UniqueName: \"kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.250584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.250564    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.250933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.250916    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.286720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.286671    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:13.347150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.347088    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerName="payload-pre-processing" containerID="cri-o://89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb" gracePeriod=25
Nov 02 14:14:13.352609 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.352576    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.352747 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.352702    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.352813 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.352767    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcmc\" (UniqueName: \"kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.352873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.352810    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.353681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.353655    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.355313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.355290    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.355942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.355916    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.375048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.375012    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcmc\" (UniqueName: \"kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc\") pod \"maas-api-e2e-praxis-b-514524-78c849544c-7ntjf\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.381205 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.381183    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:14:13.381641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.381619    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:14:13.382080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.382056    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:14:13.394878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.394855    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:13.551616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.551519    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:13.559263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.559232    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:13.595688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.590556    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"]
Nov 02 14:14:13.922806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.922775    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"]
Nov 02 14:14:13.923862 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:13.923834    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098f1ab5_ff77_47f3_b3dd_34d5aaddf9ba.slice/crio-3a76be60d34300831d866f027fa49e545497e8dc913ad1151d09bf1e6eca5272 WatchSource:0}: Error finding container 3a76be60d34300831d866f027fa49e545497e8dc913ad1151d09bf1e6eca5272: Status 404 returned error can't find the container with id 3a76be60d34300831d866f027fa49e545497e8dc913ad1151d09bf1e6eca5272
Nov 02 14:14:13.925410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:13.925394    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:14:14.610123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:14.609995    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf" event={"ID":"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba","Type":"ContainerStarted","Data":"3a76be60d34300831d866f027fa49e545497e8dc913ad1151d09bf1e6eca5272"}
Nov 02 14:14:14.892323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:14.892222    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"]
Nov 02 14:14:14.892541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:14.892489    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv" podUID="830deaaf-5d41-4ea7-8019-1db5650b80b0" containerName="maas-api" containerID="cri-o://7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5" gracePeriod=30
Nov 02 14:14:14.970688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:14.970656    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:15.325348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.317155    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:14:15.327330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.326468    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca\") pod \"830deaaf-5d41-4ea7-8019-1db5650b80b0\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") "
Nov 02 14:14:15.327330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.326557    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls\") pod \"830deaaf-5d41-4ea7-8019-1db5650b80b0\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") "
Nov 02 14:14:15.327330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.326654    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht\" (UniqueName: \"kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht\") pod \"830deaaf-5d41-4ea7-8019-1db5650b80b0\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") "
Nov 02 14:14:15.327330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.326683    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs\") pod \"830deaaf-5d41-4ea7-8019-1db5650b80b0\" (UID: \"830deaaf-5d41-4ea7-8019-1db5650b80b0\") "
Nov 02 14:14:15.331078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.330264    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca" pod "830deaaf-5d41-4ea7-8019-1db5650b80b0" (UID: "830deaaf-5d41-4ea7-8019-1db5650b80b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:15.357615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.357555    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht" pod "830deaaf-5d41-4ea7-8019-1db5650b80b0" (UID: "830deaaf-5d41-4ea7-8019-1db5650b80b0"). InnerVolumeSpecName "kube-api-access-n58ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:15.385977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.375040    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls" pod "830deaaf-5d41-4ea7-8019-1db5650b80b0" (UID: "830deaaf-5d41-4ea7-8019-1db5650b80b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:14:15.385977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.377474    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs" pod "830deaaf-5d41-4ea7-8019-1db5650b80b0" (UID: "830deaaf-5d41-4ea7-8019-1db5650b80b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:14:15.439426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.433655    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830deaaf-5d41-4ea7-8019-1db5650b80b0-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.439681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.439657    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.439830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.439816    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58ht\" (UniqueName: \"kubernetes.io/projected/830deaaf-5d41-4ea7-8019-1db5650b80b0-kube-api-access-n58ht\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.443496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.443433    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830deaaf-5d41-4ea7-8019-1db5650b80b0-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.467486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.439581    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:15.633149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.632578    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf" event={"ID":"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba","Type":"ContainerStarted","Data":"72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3"}
Nov 02 14:14:15.633149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.632833    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:15.635588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.634515    4492 generic.go:356] "Generic (PLEG): container finished" podID="830deaaf-5d41-4ea7-8019-1db5650b80b0" containerID="7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5" exitCode=0
Nov 02 14:14:15.635588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.634557    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"
Nov 02 14:14:15.635588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.634563    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv" event={"ID":"830deaaf-5d41-4ea7-8019-1db5650b80b0","Type":"ContainerDied","Data":"7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5"}
Nov 02 14:14:15.635588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.634635    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv" event={"ID":"830deaaf-5d41-4ea7-8019-1db5650b80b0","Type":"ContainerDied","Data":"cde83e23e36c0153125406f8c6dfd27eb03f66c09cd3f22c936fb1b835902bdb"}
Nov 02 14:14:15.635588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.634669    4492 scope.go:122] "RemoveContainer" containerID="7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5"
Nov 02 14:14:15.657406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.657379    4492 scope.go:122] "RemoveContainer" containerID="7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5"
Nov 02 14:14:15.657754 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:15.657731    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5\": container with ID starting with 7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5 not found: ID does not exist" containerID="7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5"
Nov 02 14:14:15.657846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.657761    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5"} err="failed to get container status \"7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5\": rpc error: code = NotFound desc = could not find container \"7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5\": container with ID starting with 7b0c2988b0e5a881dc8b0d8324fc8df41925dadebecb7c949e605953dfef5eb5 not found: ID does not exist"
Nov 02 14:14:15.677683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.677628    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf" podStartSLOduration=2.176729781 podStartE2EDuration="2.677610924s" totalImagesPullingTime="500.881143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:15.676322055 +0000 UTC m=+1020.220037650" watchObservedRunningTime="2026-11-02 14:14:15.677610924 +0000 UTC m=+1020.221326519"
Nov 02 14:14:15.693856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.693810    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:15.712127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.712091    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"]
Nov 02 14:14:15.715980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:15.715945    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-854b9e-6d4d4fdc9-c6jgv"]
Nov 02 14:14:16.624011 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:16.623972    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:14:16.799301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:16.799270    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:16.920437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:16.920006    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:17.611396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:17.611332    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830deaaf-5d41-4ea7-8019-1db5650b80b0" path="/var/lib/kubelet/pods/830deaaf-5d41-4ea7-8019-1db5650b80b0/volumes"
Nov 02 14:14:17.728633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:17.728597    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:18.409386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:18.402990    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"]
Nov 02 14:14:18.525833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:18.525800    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:18.648379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:18.647274    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8" podUID="93ae08f3-b661-4879-a3c3-52b838de1718" containerName="payload-processing" containerID="cri-o://dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434" gracePeriod=25
Nov 02 14:14:19.152800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.152773    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:14:19.217798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.217761    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume\") pod \"93ae08f3-b661-4879-a3c3-52b838de1718\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") "
Nov 02 14:14:19.217977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.217836    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials\") pod \"93ae08f3-b661-4879-a3c3-52b838de1718\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") "
Nov 02 14:14:19.218071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.217998    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs\") pod \"93ae08f3-b661-4879-a3c3-52b838de1718\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") "
Nov 02 14:14:19.218130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.218067    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume\") pod \"93ae08f3-b661-4879-a3c3-52b838de1718\" (UID: \"93ae08f3-b661-4879-a3c3-52b838de1718\") "
Nov 02 14:14:19.218256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.218232    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume" pod "93ae08f3-b661-4879-a3c3-52b838de1718" (UID: "93ae08f3-b661-4879-a3c3-52b838de1718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:19.218765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.218737    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume" pod "93ae08f3-b661-4879-a3c3-52b838de1718" (UID: "93ae08f3-b661-4879-a3c3-52b838de1718"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:19.218907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.218892    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:19.218979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.218909    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ae08f3-b661-4879-a3c3-52b838de1718-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:19.220313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.220283    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials" pod "93ae08f3-b661-4879-a3c3-52b838de1718" (UID: "93ae08f3-b661-4879-a3c3-52b838de1718"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:19.220665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.220633    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs" pod "93ae08f3-b661-4879-a3c3-52b838de1718" (UID: "93ae08f3-b661-4879-a3c3-52b838de1718"). InnerVolumeSpecName "kube-api-access-z6vfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:19.320339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.320249    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vfs\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-kube-api-access-z6vfs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:19.320339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.320286    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93ae08f3-b661-4879-a3c3-52b838de1718-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:19.743793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.743761    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:19.750539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.750501    4492 generic.go:356] "Generic (PLEG): container finished" podID="93ae08f3-b661-4879-a3c3-52b838de1718" containerID="dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434" exitCode=0
Nov 02 14:14:19.750720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.750549    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"
Nov 02 14:14:19.750720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.750579    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8" event={"ID":"93ae08f3-b661-4879-a3c3-52b838de1718","Type":"ContainerDied","Data":"dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434"}
Nov 02 14:14:19.750720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.750639    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8" event={"ID":"93ae08f3-b661-4879-a3c3-52b838de1718","Type":"ContainerDied","Data":"f0385c15ae48d09e1b70bddda43dad980765da1f9479aeddb20b15007ba020a1"}
Nov 02 14:14:19.750720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.750664    4492 scope.go:122] "RemoveContainer" containerID="dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434"
Nov 02 14:14:19.761035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.761007    4492 scope.go:122] "RemoveContainer" containerID="dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434"
Nov 02 14:14:19.761379 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:19.761337    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434\": container with ID starting with dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434 not found: ID does not exist" containerID="dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434"
Nov 02 14:14:19.761437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.761399    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434"} err="failed to get container status \"dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434\": rpc error: code = NotFound desc = could not find container \"dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434\": container with ID starting with dec076708c2b797eb9116779fccbb97b973eec5fc0a780922917d676c7e04434 not found: ID does not exist"
Nov 02 14:14:19.820507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.819260    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"]
Nov 02 14:14:19.876145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:19.876047    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864nqcv8"]
Nov 02 14:14:21.541763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:21.541725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:21.624234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:21.624194    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ae08f3-b661-4879-a3c3-52b838de1718" path="/var/lib/kubelet/pods/93ae08f3-b661-4879-a3c3-52b838de1718/volumes"
Nov 02 14:14:21.653137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:21.653108    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:14:22.041607 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:22.041564    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"]
Nov 02 14:14:22.046131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:22.046102    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64fa535b-z2ltw"]
Nov 02 14:14:22.481190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:22.481154    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:14:23.294502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.294466    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:23.407257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.407058    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw" podUID="a9db4a2c-b160-44a4-b371-d1273b825aeb" containerName="payload-processing" containerID="cri-o://3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735" gracePeriod=25
Nov 02 14:14:23.427677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.427635    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:23.584163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.584124    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2818582-06cb-4d92-add1-9ee58b523469" path="/var/lib/kubelet/pods/f2818582-06cb-4d92-add1-9ee58b523469/volumes"
Nov 02 14:14:23.715835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.715798    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:23.784882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.784852    4492 generic.go:356] "Generic (PLEG): container finished" podID="a9db4a2c-b160-44a4-b371-d1273b825aeb" containerID="3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735" exitCode=0
Nov 02 14:14:23.785067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.784929    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"
Nov 02 14:14:23.785067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.784936    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw" event={"ID":"a9db4a2c-b160-44a4-b371-d1273b825aeb","Type":"ContainerDied","Data":"3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735"}
Nov 02 14:14:23.785067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.784999    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw" event={"ID":"a9db4a2c-b160-44a4-b371-d1273b825aeb","Type":"ContainerDied","Data":"e1c4b9efa3b26ae1c07011ce694b4d1d80062c7c4bd51aa8379544c5be627412"}
Nov 02 14:14:23.785067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.785030    4492 scope.go:122] "RemoveContainer" containerID="3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735"
Nov 02 14:14:23.797105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.797075    4492 scope.go:122] "RemoveContainer" containerID="3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735"
Nov 02 14:14:23.797512 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:23.797482    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735\": container with ID starting with 3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735 not found: ID does not exist" containerID="3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735"
Nov 02 14:14:23.797617 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.797525    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735"} err="failed to get container status \"3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735\": rpc error: code = NotFound desc = could not find container \"3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735\": container with ID starting with 3c8ae88ee19db9cdada7dda8605694eb453cf91e38c1c891db0ecdad0c5d2735 not found: ID does not exist"
Nov 02 14:14:23.811426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.811396    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8\") pod \"a9db4a2c-b160-44a4-b371-d1273b825aeb\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") "
Nov 02 14:14:23.811595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.811439    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials\") pod \"a9db4a2c-b160-44a4-b371-d1273b825aeb\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") "
Nov 02 14:14:23.811595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.811528    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume\") pod \"a9db4a2c-b160-44a4-b371-d1273b825aeb\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") "
Nov 02 14:14:23.811738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.811648    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume\") pod \"a9db4a2c-b160-44a4-b371-d1273b825aeb\" (UID: \"a9db4a2c-b160-44a4-b371-d1273b825aeb\") "
Nov 02 14:14:23.811942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.811914    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume" pod "a9db4a2c-b160-44a4-b371-d1273b825aeb" (UID: "a9db4a2c-b160-44a4-b371-d1273b825aeb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:23.812074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.812053    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume" pod "a9db4a2c-b160-44a4-b371-d1273b825aeb" (UID: "a9db4a2c-b160-44a4-b371-d1273b825aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:23.812347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.812313    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:23.812434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.812382    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9db4a2c-b160-44a4-b371-d1273b825aeb-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:23.814480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.814457    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8" pod "a9db4a2c-b160-44a4-b371-d1273b825aeb" (UID: "a9db4a2c-b160-44a4-b371-d1273b825aeb"). InnerVolumeSpecName "kube-api-access-t85h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:23.814480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.814461    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials" pod "a9db4a2c-b160-44a4-b371-d1273b825aeb" (UID: "a9db4a2c-b160-44a4-b371-d1273b825aeb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:23.913591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.913558    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t85h8\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-kube-api-access-t85h8\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:23.913591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:23.913590    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9db4a2c-b160-44a4-b371-d1273b825aeb-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:24.078727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:24.078634    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:24.145399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:24.143805    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"]
Nov 02 14:14:24.159391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:24.151594    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-cfb5nwbcw"]
Nov 02 14:14:24.450545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:24.450233    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"]
Nov 02 14:14:24.451824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:24.451632    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" containerID="cri-o://75ed3f58ad0e518324e03db079f4affaa388192e7bd03321c4641c17ffbf8b46" gracePeriod=30
Nov 02 14:14:24.502736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:24.502695    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:25.318719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.318683    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:25.444318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.444280    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:25.583193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.583103    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9db4a2c-b160-44a4-b371-d1273b825aeb" path="/var/lib/kubelet/pods/a9db4a2c-b160-44a4-b371-d1273b825aeb/volumes"
Nov 02 14:14:25.769965 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.769923    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"]
Nov 02 14:14:25.770885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.770844    4492 cpu_manager.go:414] "containerMap: removing container" podUID="830deaaf-5d41-4ea7-8019-1db5650b80b0" containerName="maas-api"
Nov 02 14:14:25.770885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.770869    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830deaaf-5d41-4ea7-8019-1db5650b80b0" containerName="maas-api"
Nov 02 14:14:25.771090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.770897    4492 cpu_manager.go:414] "containerMap: removing container" podUID="93ae08f3-b661-4879-a3c3-52b838de1718" containerName="payload-processing"
Nov 02 14:14:25.771090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.770905    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ae08f3-b661-4879-a3c3-52b838de1718" containerName="payload-processing"
Nov 02 14:14:25.771090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.770947    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a9db4a2c-b160-44a4-b371-d1273b825aeb" containerName="payload-processing"
Nov 02 14:14:25.771090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.770955    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9db4a2c-b160-44a4-b371-d1273b825aeb" containerName="payload-processing"
Nov 02 14:14:25.771090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.771090    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="830deaaf-5d41-4ea7-8019-1db5650b80b0" containerName="maas-api"
Nov 02 14:14:25.771344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.771105    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ae08f3-b661-4879-a3c3-52b838de1718" containerName="payload-processing"
Nov 02 14:14:25.771344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.771120    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9db4a2c-b160-44a4-b371-d1273b825aeb" containerName="payload-processing"
Nov 02 14:14:25.776804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.776777    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.794646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.794612    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"]
Nov 02 14:14:25.799705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.799681    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-866159-openshift-default-dockercfg-jpxc8\""
Nov 02 14:14:25.818499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.818462    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:25.864254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864157    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.864456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864288    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.864456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864325    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.864569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864469    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.864569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864557    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.864680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864584    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.864738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864709    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7k9b\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.866662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864738    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.866662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.864852    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.977257    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978035    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7k9b\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978075    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978145    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978251    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978300    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978326    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978401    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978453    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:25.979426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.978478    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.005393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:25.998037    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.037394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.025282    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.037394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.025945    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.037394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.032768    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.037394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.033083    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.037394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.036701    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.086405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.069717    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7k9b\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.086405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.075779    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.112292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.112220    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.393522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.393478    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:26.728904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.728865    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"]
Nov 02 14:14:26.730213 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:26.730187    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da14694_3a1f_4327_aa17_77ec3cd9549f.slice/crio-1e01de09952d8c4721a81e71be304659476931d662b6bb5df53e96a575fcaacf WatchSource:0}: Error finding container 1e01de09952d8c4721a81e71be304659476931d662b6bb5df53e96a575fcaacf: Status 404 returned error can't find the container with id 1e01de09952d8c4721a81e71be304659476931d662b6bb5df53e96a575fcaacf
Nov 02 14:14:26.732781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.732747    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:14:26.732879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.732832    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:14:26.732947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.732874    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:14:26.826401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.825666    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" event={"ID":"0da14694-3a1f-4327-aa17-77ec3cd9549f","Type":"ContainerStarted","Data":"f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"}
Nov 02 14:14:26.826401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.825739    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" event={"ID":"0da14694-3a1f-4327-aa17-77ec3cd9549f","Type":"ContainerStarted","Data":"1e01de09952d8c4721a81e71be304659476931d662b6bb5df53e96a575fcaacf"}
Nov 02 14:14:26.868995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:26.868838    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podStartSLOduration=1.8688171279999999 podStartE2EDuration="1.868817128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:26.866860096 +0000 UTC m=+1031.410575691" watchObservedRunningTime="2026-11-02 14:14:26.868817128 +0000 UTC m=+1031.412532723"
Nov 02 14:14:27.342861 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:27.342830    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:27.399379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:27.397663    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:27.423382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:27.417897    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:27.847077 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:27.847048    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:27.848148 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:27.848128    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:14:28.177381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:28.173455    4492 patch_prober.go:29] interesting pod/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:14:28.177381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:28.173558    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:14:29.925452 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:29.917516    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:29.958686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:29.958652    4492 generic.go:356] "Generic (PLEG): container finished" podID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerID="75ed3f58ad0e518324e03db079f4affaa388192e7bd03321c4641c17ffbf8b46" exitCode=0
Nov 02 14:14:29.959155 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:29.959007    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" event={"ID":"0270c6f0-c594-4e27-abc8-2e192ca43cf0","Type":"ContainerDied","Data":"75ed3f58ad0e518324e03db079f4affaa388192e7bd03321c4641c17ffbf8b46"}
Nov 02 14:14:29.959458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:29.959342    4492 scope.go:122] "RemoveContainer" containerID="6e8819583d425ab328c4154c866aebd0f394b924410dc17d441d897794cec1fc"
Nov 02 14:14:30.614492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:30.604471    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:30.764589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:30.737004    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:30.938432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:30.931931    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.011871    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.011984    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012110    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012192    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012255    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012283    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012405    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012462    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.017818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.012499    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs\") pod \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\" (UID: \"0270c6f0-c594-4e27-abc8-2e192ca43cf0\") "
Nov 02 14:14:31.020091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.020057    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r" event={"ID":"0270c6f0-c594-4e27-abc8-2e192ca43cf0","Type":"ContainerDied","Data":"2b7eeb4dfdb7f2660f3eaa3c0706ab2b4069e11ec7992509efe7165ee47336a1"}
Nov 02 14:14:31.020496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.020471    4492 scope.go:122] "RemoveContainer" containerID="75ed3f58ad0e518324e03db079f4affaa388192e7bd03321c4641c17ffbf8b46"
Nov 02 14:14:31.020804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.020788    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"
Nov 02 14:14:31.029585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.025596    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:31.049886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.048610    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:31.049886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.048904    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:31.049886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.049203    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:31.049886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.049309    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:31.049886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.049609    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:31.068019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.067827    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.068019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.067865    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.068019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.067903    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.068019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.067923    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.068019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.067938    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.068019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.067952    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.087985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.070263    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:31.087985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.084604    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:31.166425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.162891    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:31.166425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.163151    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h" pod "0270c6f0-c594-4e27-abc8-2e192ca43cf0" (UID: "0270c6f0-c594-4e27-abc8-2e192ca43cf0"). InnerVolumeSpecName "kube-api-access-dq24h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:31.221682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.221649    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.221682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.221690    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq24h\" (UniqueName: \"kubernetes.io/projected/0270c6f0-c594-4e27-abc8-2e192ca43cf0-kube-api-access-dq24h\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.221971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.221709    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0270c6f0-c594-4e27-abc8-2e192ca43cf0-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.368306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.368265    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"]
Nov 02 14:14:31.376459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.376410    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-854b9e-openshift-default-7dd9bd658c-5dg9r"]
Nov 02 14:14:31.582328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:31.582254    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" path="/var/lib/kubelet/pods/0270c6f0-c594-4e27-abc8-2e192ca43cf0/volumes"
Nov 02 14:14:32.107554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.107508    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:32.233048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.233007    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:32.286277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.286237    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"]
Nov 02 14:14:32.286902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.286888    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy"
Nov 02 14:14:32.286941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.286904    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy"
Nov 02 14:14:32.287028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.287018    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy"
Nov 02 14:14:32.287083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.287034    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy"
Nov 02 14:14:32.291734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.291713    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.306526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.306501    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-866159\""
Nov 02 14:14:32.316044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.316011    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-866159-dockercfg-sf9kv\""
Nov 02 14:14:32.322529 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.322495    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"]
Nov 02 14:14:32.436437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.435426    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"]
Nov 02 14:14:32.436612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.436518    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy"
Nov 02 14:14:32.436612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.436541    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0270c6f0-c594-4e27-abc8-2e192ca43cf0" containerName="istio-proxy"
Nov 02 14:14:32.444377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.443247    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9z9\" (UniqueName: \"kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9\") pod \"payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.444377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.443438    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.484463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.463743    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.571105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.486847    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"]
Nov 02 14:14:32.610443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.500471    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:14:32.610443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.549204    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.610443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.595000    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcdw\" (UniqueName: \"kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw\") pod \"payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.610443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.595061    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9z9\" (UniqueName: \"kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9\") pod \"payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.610443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.595144    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.610443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.568004    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.641802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.641778    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9z9\" (UniqueName: \"kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9\") pod \"payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:32.696924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.696838    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcdw\" (UniqueName: \"kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw\") pod \"payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.697081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.696936    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.697638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.697617    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.720542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.720508    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcdw\" (UniqueName: \"kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw\") pod \"payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.803934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.802680    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:32.906812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:32.906779    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:33.112873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.112702    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:33.497129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.497090    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"]
Nov 02 14:14:33.501153 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:33.501106    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077b9507_47f3_4c34_82c7_27f5a954feb8.slice/crio-7b416fbc3c2cf4a513d162c5448f55f6bbd96c774b65e48d0c68276af932bbe4 WatchSource:0}: Error finding container 7b416fbc3c2cf4a513d162c5448f55f6bbd96c774b65e48d0c68276af932bbe4: Status 404 returned error can't find the container with id 7b416fbc3c2cf4a513d162c5448f55f6bbd96c774b65e48d0c68276af932bbe4
Nov 02 14:14:33.610662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.610550    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"]
Nov 02 14:14:33.614344 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:33.614314    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8e9a7b_0c64_4177_8f58_8be6670c3f2f.slice/crio-4716e5ca7963412a0f55181264511d072b06f5648763a0f900b0ccb4d08c4260 WatchSource:0}: Error finding container 4716e5ca7963412a0f55181264511d072b06f5648763a0f900b0ccb4d08c4260: Status 404 returned error can't find the container with id 4716e5ca7963412a0f55181264511d072b06f5648763a0f900b0ccb4d08c4260
Nov 02 14:14:33.683951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.683920    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:14:33.770533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.770481    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume\") pod \"5e86c391-5506-479e-b9d4-66f8a9b58c68\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") "
Nov 02 14:14:33.770731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.770634    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk\" (UniqueName: \"kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk\") pod \"5e86c391-5506-479e-b9d4-66f8a9b58c68\" (UID: \"5e86c391-5506-479e-b9d4-66f8a9b58c68\") "
Nov 02 14:14:33.770919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.770894    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume" pod "5e86c391-5506-479e-b9d4-66f8a9b58c68" (UID: "5e86c391-5506-479e-b9d4-66f8a9b58c68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:33.771168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.771150    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86c391-5506-479e-b9d4-66f8a9b58c68-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:33.773335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.773299    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk" pod "5e86c391-5506-479e-b9d4-66f8a9b58c68" (UID: "5e86c391-5506-479e-b9d4-66f8a9b58c68"). InnerVolumeSpecName "kube-api-access-wznqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:33.872653 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:33.872574    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznqk\" (UniqueName: \"kubernetes.io/projected/5e86c391-5506-479e-b9d4-66f8a9b58c68-kube-api-access-wznqk\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:34.129453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.129342    4492 generic.go:356] "Generic (PLEG): container finished" podID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerID="89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb" exitCode=0
Nov 02 14:14:34.129453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.129417    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"
Nov 02 14:14:34.129453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.129425    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" event={"ID":"5e86c391-5506-479e-b9d4-66f8a9b58c68","Type":"ContainerDied","Data":"89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb"}
Nov 02 14:14:34.130061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.129475    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b" event={"ID":"5e86c391-5506-479e-b9d4-66f8a9b58c68","Type":"ContainerDied","Data":"8a805a3979803f4d9aa3175b781e6e1980a3e3df66934d41da01315f961674fb"}
Nov 02 14:14:34.130061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.129496    4492 scope.go:122] "RemoveContainer" containerID="89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb"
Nov 02 14:14:34.131405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.131348    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" event={"ID":"077b9507-47f3-4c34-82c7-27f5a954feb8","Type":"ContainerStarted","Data":"f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917"}
Nov 02 14:14:34.131525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.131432    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" event={"ID":"077b9507-47f3-4c34-82c7-27f5a954feb8","Type":"ContainerStarted","Data":"7b416fbc3c2cf4a513d162c5448f55f6bbd96c774b65e48d0c68276af932bbe4"}
Nov 02 14:14:34.131525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.131462    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:34.150769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.150719    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" event={"ID":"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f","Type":"ContainerStarted","Data":"c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52"}
Nov 02 14:14:34.151085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.151038    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" event={"ID":"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f","Type":"ContainerStarted","Data":"4716e5ca7963412a0f55181264511d072b06f5648763a0f900b0ccb4d08c4260"}
Nov 02 14:14:34.152261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.152238    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:34.169340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.169311    4492 scope.go:122] "RemoveContainer" containerID="89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb"
Nov 02 14:14:34.169759 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:34.169730    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb\": container with ID starting with 89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb not found: ID does not exist" containerID="89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb"
Nov 02 14:14:34.169822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.169774    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb"} err="failed to get container status \"89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb\": rpc error: code = NotFound desc = could not find container \"89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb\": container with ID starting with 89d3b5a5acf1279572b230c7a57a4a281c699843d7f7f66f76ee5e86eed4e0eb not found: ID does not exist"
Nov 02 14:14:34.193776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.193710    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" podStartSLOduration=2.193693459 podStartE2EDuration="2.193693459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:34.190380446 +0000 UTC m=+1038.734096040" watchObservedRunningTime="2026-11-02 14:14:34.193693459 +0000 UTC m=+1038.737409055"
Nov 02 14:14:34.235516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.235451    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" podStartSLOduration=2.235429925 podStartE2EDuration="2.235429925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:34.230623252 +0000 UTC m=+1038.774338846" watchObservedRunningTime="2026-11-02 14:14:34.235429925 +0000 UTC m=+1038.779145519"
Nov 02 14:14:34.275266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.275234    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"]
Nov 02 14:14:34.278922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:34.278893    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-854b9e-75c9b5dcdc-h8j4b"]
Nov 02 14:14:35.028500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:35.028468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:35.582388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:35.582339    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" path="/var/lib/kubelet/pods/5e86c391-5506-479e-b9d4-66f8a9b58c68/volumes"
Nov 02 14:14:36.717935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:36.717888    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:38.368945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:38.368910    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:38.517498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:38.517460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:40.244005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:40.243955    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:42.041006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:42.040972    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:43.271182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:43.271148    4492 generic.go:356] "Generic (PLEG): container finished" podID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerID="87581e28fba08cd11cdff6a2c24e1d26b44448e651bc4709c998c79e1a0b2e7f" exitCode=0
Nov 02 14:14:43.271759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:43.271224    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" event={"ID":"00dcb0fb-68a0-4605-b953-5426cb18014a","Type":"ContainerDied","Data":"87581e28fba08cd11cdff6a2c24e1d26b44448e651bc4709c998c79e1a0b2e7f"}
Nov 02 14:14:43.271837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:43.271774    4492 scope.go:122] "RemoveContainer" containerID="87581e28fba08cd11cdff6a2c24e1d26b44448e651bc4709c998c79e1a0b2e7f"
Nov 02 14:14:43.273287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:43.273257    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:14:43.273386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:43.273332    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:14:43.273434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:43.273390    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:14:44.064940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:44.064887    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:44.281657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:44.281619    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" event={"ID":"00dcb0fb-68a0-4605-b953-5426cb18014a","Type":"ContainerStarted","Data":"db613db5ded6c87ae233b5da985d44c9ff6b9bf90c235e4655cd033753fc4403"}
Nov 02 14:14:44.759742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:44.759702    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:14:44.760951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:44.760909    4492 patch_prober.go:29] interesting pod/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 14:14:44.761098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:44.760964    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 14:14:45.184886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.184856    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:14:45.759870 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.759833    4492 patch_prober.go:29] interesting pod/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 14:14:45.760414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.759899    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 14:14:45.787970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.787927    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"]
Nov 02 14:14:45.788709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.788685    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerName="payload-pre-processing"
Nov 02 14:14:45.788836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.788709    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerName="payload-pre-processing"
Nov 02 14:14:45.788902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.788881    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e86c391-5506-479e-b9d4-66f8a9b58c68" containerName="payload-pre-processing"
Nov 02 14:14:45.792855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.792831    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.819115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.819062    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"]
Nov 02 14:14:45.822442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.822414    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-866159-serving-cert\""
Nov 02 14:14:45.886541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.886506    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.886755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.886582    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.886755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.886659    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.886880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.886754    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frndx\" (UniqueName: \"kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.914885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.914854    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:45.987494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.987451    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frndx\" (UniqueName: \"kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.987704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.987533    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.987704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.987582    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.987704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.987648    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.988418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.988302    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.990335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.990313    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:45.990498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:45.990478    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:46.050299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.050137    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:46.062920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.062874    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frndx\" (UniqueName: \"kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx\") pod \"maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:46.110904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.110617    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:46.134345 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.134315    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:14:46.193195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.193165    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:14:46.474509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.474475    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"]
Nov 02 14:14:46.480333 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:46.480304    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb27a56f_6686_4998_92d8_43e02b245981.slice/crio-f8148c20cf4f1941cb0f4651b50cbf2324dddf75ae011bbb4f6f0c4b842ec2c8 WatchSource:0}: Error finding container f8148c20cf4f1941cb0f4651b50cbf2324dddf75ae011bbb4f6f0c4b842ec2c8: Status 404 returned error can't find the container with id f8148c20cf4f1941cb0f4651b50cbf2324dddf75ae011bbb4f6f0c4b842ec2c8
Nov 02 14:14:46.518416 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:46.518379    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:14:46.760234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.760132    4492 patch_prober.go:29] interesting pod/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 14:14:46.760234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:46.760221    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 14:14:47.305816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.305779    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:47.314338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.313551    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9" event={"ID":"db27a56f-6686-4998-92d8-43e02b245981","Type":"ContainerStarted","Data":"3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c"}
Nov 02 14:14:47.314338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.314160    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9" event={"ID":"db27a56f-6686-4998-92d8-43e02b245981","Type":"ContainerStarted","Data":"f8148c20cf4f1941cb0f4651b50cbf2324dddf75ae011bbb4f6f0c4b842ec2c8"}
Nov 02 14:14:47.314338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.314192    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:47.366137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.365286    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9" podStartSLOduration=1.944578354 podStartE2EDuration="2.365245258s" totalImagesPullingTime="420.666904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:47.362819255 +0000 UTC m=+1051.906534873" watchObservedRunningTime="2026-11-02 14:14:47.365245258 +0000 UTC m=+1051.908960853"
Nov 02 14:14:47.786381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.760810    4492 patch_prober.go:29] interesting pod/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 14:14:47.786381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:47.760899    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 14:14:48.054450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:48.051807    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:48.761889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:48.761849    4492 patch_prober.go:29] interesting pod/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 02 14:14:48.762249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:48.762214    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 02 14:14:49.765675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:49.765629    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:14:49.766987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:49.766950    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:14:49.817592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:49.817559    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:51.611752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:51.611709    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:52.489036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.488999    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"]
Nov 02 14:14:52.489302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.489271    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" containerID="cri-o://135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c" gracePeriod=30
Nov 02 14:14:52.499028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.498992    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"]
Nov 02 14:14:52.499263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.499239    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy" containerID="cri-o://dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5" gracePeriod=30
Nov 02 14:14:52.504403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.504374    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"]
Nov 02 14:14:52.504834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.504802    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy" containerID="cri-o://3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827" gracePeriod=30
Nov 02 14:14:52.511536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.511508    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"]
Nov 02 14:14:52.511716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.511697    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy" containerID="cri-o://f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea" gracePeriod=30
Nov 02 14:14:52.595410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:52.595374    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:53.397931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:53.397894    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:14:53.675201 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:53.675022    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:55.632165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:55.632136    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:56.272534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:56.272501    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:56.385519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:56.385468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:57.505660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.505621    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-clxsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:14:57.506264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.506233    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:14:57.628440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.628331    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"]
Nov 02 14:14:57.671685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.647703    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.671685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.669163    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"]
Nov 02 14:14:57.675576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.669346    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-ddd47dcbd9ba\""
Nov 02 14:14:57.729734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.729696    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.730075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.730047    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj4l5\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.730257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.730240    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.730413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.730393    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.826932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.826885    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:57.832293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.832251    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj4l5\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.832473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.832347    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.832473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.832446    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.836375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.832629    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.873212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.871394    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.875024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.874786    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.875710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.875649    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:57.985680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:57.978076    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj4l5\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:58.096176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.096144    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:14:58.096335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.096316    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:58.103589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.103562    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:14:58.111855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.111782    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:14:58.124379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.124343    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:14:58.189336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189287    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.189336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189349    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189414    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189463    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189541    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189593    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189656    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189685    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.189748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189712    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189762    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189793    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189819    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189859    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189871    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189900    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189969    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.189976    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190024    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190067    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.190132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190103    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190150    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190161    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190229    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190235    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190275    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190533    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190656    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.190719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190693    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.191125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190777    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.191125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190824    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert\") pod \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\" (UID: \"1fa1ff1d-e21b-40d7-8607-3580f21010d4\") "
Nov 02 14:14:58.191125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190857    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token\") pod \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\" (UID: \"93c11fbf-0a2c-4de7-a993-5cd1e9427a93\") "
Nov 02 14:14:58.191125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.190876    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs\") pod \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\" (UID: \"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa\") "
Nov 02 14:14:58.191125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191055    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:58.191404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191219    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191663    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191693    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191715    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191730    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191744    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191757    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191773    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191786    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.191843    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.192114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.192007    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.192719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.192255    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:58.192719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.192509    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.192830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.192789    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:58.193733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.193464    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.193904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.193814    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.194512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.194193    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:58.195387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.195328    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.197869    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "kube-api-access-x7nbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.197915    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199581    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "kube-api-access-m7w6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199655    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199696    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199705    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token" pod "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" (UID: "737bb7e1-871d-43d6-80b8-7a22ab0c1ffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199744    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199858    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.199891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.199874    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy" pod "1fa1ff1d-e21b-40d7-8607-3580f21010d4" (UID: "1fa1ff1d-e21b-40d7-8607-3580f21010d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.201943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.201901    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n" pod "93c11fbf-0a2c-4de7-a993-5cd1e9427a93" (UID: "93c11fbf-0a2c-4de7-a993-5cd1e9427a93"). InnerVolumeSpecName "kube-api-access-64t5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.292812    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.292865    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.292900    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.292976    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293002    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293031    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293055    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293085    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293123    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket\") pod \"ebb3a350-5224-4483-99c2-e3dbc240d43e\" (UID: \"ebb3a350-5224-4483-99c2-e3dbc240d43e\") "
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293424    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64t5n\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-kube-api-access-64t5n\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293436    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293445    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293453    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293463    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7nbp\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-kube-api-access-x7nbp\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293471    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293480    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293488    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293497    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293504    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293512    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293520    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293528    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293536    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293544    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293551    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293558    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa1ff1d-e21b-40d7-8607-3580f21010d4-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293567    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293574    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7w6r\" (UniqueName: \"kubernetes.io/projected/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa-kube-api-access-m7w6r\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293582    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c11fbf-0a2c-4de7-a993-5cd1e9427a93-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.293788    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.294337    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.294398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.294399    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:58.296081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.294653    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.296081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.294844    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.297006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.296584    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "kube-api-access-qhsbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.297610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.297579    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:58.297729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.297601    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:58.297973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.297952    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo" pod "ebb3a350-5224-4483-99c2-e3dbc240d43e" (UID: "ebb3a350-5224-4483-99c2-e3dbc240d43e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394253    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhsbf\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-kube-api-access-qhsbf\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394283    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394296    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb3a350-5224-4483-99c2-e3dbc240d43e-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394304    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394311    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394319    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394328    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394338    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb3a350-5224-4483-99c2-e3dbc240d43e-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.394367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.394346    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb3a350-5224-4483-99c2-e3dbc240d43e-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:14:58.506211 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.504423    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"]
Nov 02 14:14:58.584824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.584552    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:58.595967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.595933    4492 generic.go:356] "Generic (PLEG): container finished" podID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerID="dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5" exitCode=0
Nov 02 14:14:58.596101 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.596082    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" event={"ID":"ebb3a350-5224-4483-99c2-e3dbc240d43e","Type":"ContainerDied","Data":"dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5"}
Nov 02 14:14:58.596166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.596139    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c" event={"ID":"ebb3a350-5224-4483-99c2-e3dbc240d43e","Type":"ContainerDied","Data":"f4e0a71c4440d7260df32818fdb848eb0d3dae59f16f49a8c41bd73b1ca2a761"}
Nov 02 14:14:58.596202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.596167    4492 scope.go:122] "RemoveContainer" containerID="dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5"
Nov 02 14:14:58.596335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.596322    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"
Nov 02 14:14:58.606171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.603069    4492 generic.go:356] "Generic (PLEG): container finished" podID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerID="135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c" exitCode=0
Nov 02 14:14:58.606171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.603417    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" event={"ID":"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa","Type":"ContainerDied","Data":"135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c"}
Nov 02 14:14:58.606171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.603568    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz" event={"ID":"737bb7e1-871d-43d6-80b8-7a22ab0c1ffa","Type":"ContainerDied","Data":"a96095d6c6efd651ecf64fd0b35c9d3a85a95aee38869ddcff5b117c26e09060"}
Nov 02 14:14:58.606171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.603765    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"
Nov 02 14:14:58.610486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.610295    4492 scope.go:122] "RemoveContainer" containerID="dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5"
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.610630    4492 generic.go:356] "Generic (PLEG): container finished" podID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerID="f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea" exitCode=0
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.610692    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" event={"ID":"93c11fbf-0a2c-4de7-a993-5cd1e9427a93","Type":"ContainerDied","Data":"f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea"}
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.610727    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz" event={"ID":"93c11fbf-0a2c-4de7-a993-5cd1e9427a93","Type":"ContainerDied","Data":"331e5581b1631debaabcfbd4c676af868e0293883f7a1cd3a0c9a463d847265b"}
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.610784    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:58.612919    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5\": container with ID starting with dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5 not found: ID does not exist" containerID="dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5"
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.612957    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5"} err="failed to get container status \"dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5\": rpc error: code = NotFound desc = could not find container \"dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5\": container with ID starting with dda40ef5bf08b985e3dcc2e1391659b8531dc05b341da6b9058d7fb7683951d5 not found: ID does not exist"
Nov 02 14:14:58.614314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.612984    4492 scope.go:122] "RemoveContainer" containerID="135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c"
Nov 02 14:14:58.615502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.615270    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8" event={"ID":"4a3d6957-6141-4a18-82e0-d5e83ceb918c","Type":"ContainerStarted","Data":"2a7bed3e58fa1a5abe06f457822122e62ebc8d73f1e2a11ca6bdec336783bf92"}
Nov 02 14:14:58.622869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.621844    4492 generic.go:356] "Generic (PLEG): container finished" podID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerID="3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827" exitCode=0
Nov 02 14:14:58.622869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.622330    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" event={"ID":"1fa1ff1d-e21b-40d7-8607-3580f21010d4","Type":"ContainerDied","Data":"3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827"}
Nov 02 14:14:58.622869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.622453    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk" event={"ID":"1fa1ff1d-e21b-40d7-8607-3580f21010d4","Type":"ContainerDied","Data":"9211c2dc29b7a28a239cda306e386b4b41ba97d2fcace59856d8819f599f7df0"}
Nov 02 14:14:58.625544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.623107    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"
Nov 02 14:14:58.788833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.783224    4492 scope.go:122] "RemoveContainer" containerID="3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"
Nov 02 14:14:58.894461 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.883723    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"]
Nov 02 14:14:58.901469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.901054    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-clxsz"]
Nov 02 14:14:58.908804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.908762    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"]
Nov 02 14:14:58.910580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.909395    4492 scope.go:122] "RemoveContainer" containerID="135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c"
Nov 02 14:14:58.910580 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:58.910334    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c\": container with ID starting with 135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c not found: ID does not exist" containerID="135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c"
Nov 02 14:14:58.910580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.910405    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c"} err="failed to get container status \"135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c\": rpc error: code = NotFound desc = could not find container \"135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c\": container with ID starting with 135ec93804f83530a2c6c1056be25f9f1254d28271bf61ab897ea37ccf8ad03c not found: ID does not exist"
Nov 02 14:14:58.910580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.910468    4492 scope.go:122] "RemoveContainer" containerID="3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"
Nov 02 14:14:58.911038 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:58.911015    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac\": container with ID starting with 3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac not found: ID does not exist" containerID="3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"
Nov 02 14:14:58.911112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.911045    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac"} err="failed to get container status \"3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac\": rpc error: code = NotFound desc = could not find container \"3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac\": container with ID starting with 3a2464bbe068ef8046b4ca964b53d11a6f7f6c545564778fb592a15ae2c400ac not found: ID does not exist"
Nov 02 14:14:58.911112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.911066    4492 scope.go:122] "RemoveContainer" containerID="f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea"
Nov 02 14:14:58.915589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.914889    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"]
Nov 02 14:14:58.916143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916122    4492 cpu_manager.go:414] "containerMap: removing container" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy"
Nov 02 14:14:58.916192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916149    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy"
Nov 02 14:14:58.916192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916168    4492 cpu_manager.go:414] "containerMap: removing container" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy"
Nov 02 14:14:58.916192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916178    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy"
Nov 02 14:14:58.916296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916194    4492 cpu_manager.go:414] "containerMap: removing container" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy"
Nov 02 14:14:58.916296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916202    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy"
Nov 02 14:14:58.916296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916217    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy"
Nov 02 14:14:58.916296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916225    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy"
Nov 02 14:14:58.916296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916269    4492 cpu_manager.go:414] "containerMap: removing container" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy"
Nov 02 14:14:58.916296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916277    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy"
Nov 02 14:14:58.916499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916454    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy"
Nov 02 14:14:58.916499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916469    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" containerName="istio-proxy"
Nov 02 14:14:58.916499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916485    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" containerName="istio-proxy"
Nov 02 14:14:58.916499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:58.916498    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" containerName="istio-proxy"
Nov 02 14:14:59.078866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.078841    4492 scope.go:122] "RemoveContainer" containerID="f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea"
Nov 02 14:14:59.105397 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:59.101507    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea\": container with ID starting with f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea not found: ID does not exist" containerID="f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea"
Nov 02 14:14:59.105397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.101554    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea"} err="failed to get container status \"f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea\": rpc error: code = NotFound desc = could not find container \"f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea\": container with ID starting with f57a019f7899c6435b37ba0f7fe3b1688c8565995487f5a93dd60b8ed56f9fea not found: ID does not exist"
Nov 02 14:14:59.105397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.101586    4492 scope.go:122] "RemoveContainer" containerID="3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827"
Nov 02 14:14:59.138035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.126326    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.153058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.152973    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bf0b0d\""
Nov 02 14:14:59.154088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.154065    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"payload-processing-external-model-e2e--d319af1f-dockercfg-wq4bm\""
Nov 02 14:14:59.154292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.154278    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"routing-overlay\""
Nov 02 14:14:59.154470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.154455    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"agc-cred-e29452e73ed3\""
Nov 02 14:14:59.154657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.154643    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"kube-root-ca.crt\""
Nov 02 14:14:59.154869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.154854    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bf0b0d\"/\"openshift-service-ca.crt\""
Nov 02 14:14:59.173400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.168341    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-b8n6c"]
Nov 02 14:14:59.173400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.168417    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"]
Nov 02 14:14:59.173400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.168467    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"]
Nov 02 14:14:59.173400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.168485    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4nbcz"]
Nov 02 14:14:59.173400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.168502    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"]
Nov 02 14:14:59.173400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.168518    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-bxtwk"]
Nov 02 14:14:59.192758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.192719    4492 scope.go:122] "RemoveContainer" containerID="3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827"
Nov 02 14:14:59.193163 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:14:59.193102    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827\": container with ID starting with 3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827 not found: ID does not exist" containerID="3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827"
Nov 02 14:14:59.193163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.193134    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827"} err="failed to get container status \"3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827\": rpc error: code = NotFound desc = could not find container \"3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827\": container with ID starting with 3fa8a4253eb8d9f1ab96efd4ba8dbece74b202c1877161e4345e2758c2820827 not found: ID does not exist"
Nov 02 14:14:59.257456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.257404    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.257672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.257528    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvpr\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.257733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.257667    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.257786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.257741    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.358998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.358962    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.359232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.359077    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.359232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.359111    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvpr\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.359232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.359185    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.360069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.359978    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.360069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.359989    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.362651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.362451    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.381034 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.381008    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvpr\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr\") pod \"payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") " pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.495694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.495657    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:14:59.589723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.589684    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa1ff1d-e21b-40d7-8607-3580f21010d4" path="/var/lib/kubelet/pods/1fa1ff1d-e21b-40d7-8607-3580f21010d4/volumes"
Nov 02 14:14:59.591719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.591689    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" path="/var/lib/kubelet/pods/737bb7e1-871d-43d6-80b8-7a22ab0c1ffa/volumes"
Nov 02 14:14:59.592637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.592614    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c11fbf-0a2c-4de7-a993-5cd1e9427a93" path="/var/lib/kubelet/pods/93c11fbf-0a2c-4de7-a993-5cd1e9427a93/volumes"
Nov 02 14:14:59.593936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.593902    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb3a350-5224-4483-99c2-e3dbc240d43e" path="/var/lib/kubelet/pods/ebb3a350-5224-4483-99c2-e3dbc240d43e/volumes"
Nov 02 14:14:59.637827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.637774    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8" event={"ID":"4a3d6957-6141-4a18-82e0-d5e83ceb918c","Type":"ContainerStarted","Data":"19a887f44f022226f1c21d85ef10d027c44cefe391a18596f4980b0143172b0c"}
Nov 02 14:14:59.638895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.638867    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:14:59.639861 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.639777    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:14:59.684683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.684613    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8" podStartSLOduration=2.6845883329999998 podStartE2EDuration="2.684588333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:59.676564243 +0000 UTC m=+1064.220279838" watchObservedRunningTime="2026-11-02 14:14:59.684588333 +0000 UTC m=+1064.228303927"
Nov 02 14:14:59.815699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:14:59.815677    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"]
Nov 02 14:14:59.818149 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:14:59.818120    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf067613_99dd_4278_aef5_a64464655c4c.slice/crio-8ce8834cf8dde1817e8c5692260cb13f85d8b715614096d2f64fa58ea3b84b16 WatchSource:0}: Error finding container 8ce8834cf8dde1817e8c5692260cb13f85d8b715614096d2f64fa58ea3b84b16: Status 404 returned error can't find the container with id 8ce8834cf8dde1817e8c5692260cb13f85d8b715614096d2f64fa58ea3b84b16
Nov 02 14:15:00.170188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.170103    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"]
Nov 02 14:15:00.170845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.170827    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="737bb7e1-871d-43d6-80b8-7a22ab0c1ffa" containerName="istio-proxy"
Nov 02 14:15:00.174672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.174643    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.211698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.211497    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"]
Nov 02 14:15:00.225403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.223799    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"]
Nov 02 14:15:00.225403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.223973    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.250742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.250717    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"]
Nov 02 14:15:00.250883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.250760    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"]
Nov 02 14:15:00.250883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.250778    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"]
Nov 02 14:15:00.250883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.250879    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.279845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.279723    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg6r\" (UniqueName: \"kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r\") pod \"maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.279845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.279778    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.280089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.279823    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.280150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.280100    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6j55\" (UniqueName: \"kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.281575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.281548    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:15:00.381249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.381203    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.381249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.381252    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdjl\" (UniqueName: \"kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl\") pod \"maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.381521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.381452    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg6r\" (UniqueName: \"kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r\") pod \"maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.381521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.381500    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.381626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.381543    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.381626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.381605    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6j55\" (UniqueName: \"kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.382234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.382212    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.382417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.382395    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.407804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.407769    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6j55\" (UniqueName: \"kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55\") pod \"maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.408079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.408056    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg6r\" (UniqueName: \"kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r\") pod \"maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.483030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.482987    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.483402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.483346    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdjl\" (UniqueName: \"kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl\") pod \"maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.483961 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.483937    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.492047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.492018    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:00.505385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.505320    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdjl\" (UniqueName: \"kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl\") pod \"maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.556686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.555641    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:00.567557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.567233    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:00.682999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.682793    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm" event={"ID":"cf067613-99dd-4278-aef5-a64464655c4c","Type":"ContainerStarted","Data":"0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d"}
Nov 02 14:15:00.682999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.682891    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm" event={"ID":"cf067613-99dd-4278-aef5-a64464655c4c","Type":"ContainerStarted","Data":"8ce8834cf8dde1817e8c5692260cb13f85d8b715614096d2f64fa58ea3b84b16"}
Nov 02 14:15:00.682999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.682956    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:15:00.723332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.723265    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm" podStartSLOduration=2.723244804 podStartE2EDuration="2.723244804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:00.719152987 +0000 UTC m=+1065.262868582" watchObservedRunningTime="2026-11-02 14:15:00.723244804 +0000 UTC m=+1065.266960399"
Nov 02 14:15:00.817615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.817539    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"]
Nov 02 14:15:00.935416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.926815    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"]
Nov 02 14:15:00.949349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:00.949315    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"]
Nov 02 14:15:00.956595 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:15:00.956568    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d74a3fa_03e9_497c_a3ab_8b3d011ad603.slice/crio-8ce6ac2d5a353701a89f56b9d3ff69a247d6143894455f131ff98be362101646 WatchSource:0}: Error finding container 8ce6ac2d5a353701a89f56b9d3ff69a247d6143894455f131ff98be362101646: Status 404 returned error can't find the container with id 8ce6ac2d5a353701a89f56b9d3ff69a247d6143894455f131ff98be362101646
Nov 02 14:15:01.543406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.543343    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:01.690387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.690321    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerStarted","Data":"265b6939e49e3f35aac6a32515a9893ddad5bf71a43c27d7f9127d160b0be1eb"}
Nov 02 14:15:01.690808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.690408    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerStarted","Data":"71bf9334703c62743ff3c753320778eb56f78fe7ac160ddfefe8b01b36ec519c"}
Nov 02 14:15:01.692196 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.692162    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerStarted","Data":"27d21dfbefe7e4493f2296c03b35cf814bb3100cfad48ff0f229b483203a148b"}
Nov 02 14:15:01.692342 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.692212    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerStarted","Data":"52f99a288baff1468b39bcd4010d39ff4059ed34284dc8c02e4c5ff951a7685e"}
Nov 02 14:15:01.693988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.693962    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerStarted","Data":"2a05031a00f8d0bc81c83538b58699c02b4be4dd2370fa19e1af0c9902bda857"}
Nov 02 14:15:01.694105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.693999    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerStarted","Data":"8ce6ac2d5a353701a89f56b9d3ff69a247d6143894455f131ff98be362101646"}
Nov 02 14:15:01.730002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.729947    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" podStartSLOduration=1.72993201 podStartE2EDuration="1.72993201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.725651994 +0000 UTC m=+1066.269367588" watchObservedRunningTime="2026-11-02 14:15:01.72993201 +0000 UTC m=+1066.273647646"
Nov 02 14:15:01.770520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.770445    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" podStartSLOduration=1.770422017 podStartE2EDuration="1.770422017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.764139073 +0000 UTC m=+1066.307854669" watchObservedRunningTime="2026-11-02 14:15:01.770422017 +0000 UTC m=+1066.314137612"
Nov 02 14:15:01.805775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:01.805705    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" podStartSLOduration=1.805684496 podStartE2EDuration="1.805684496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.801841882 +0000 UTC m=+1066.345557477" watchObservedRunningTime="2026-11-02 14:15:01.805684496 +0000 UTC m=+1066.349400092"
Nov 02 14:15:02.703272 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.703233    4492 generic.go:356] "Generic (PLEG): container finished" podID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerID="2a05031a00f8d0bc81c83538b58699c02b4be4dd2370fa19e1af0c9902bda857" exitCode=60
Nov 02 14:15:02.703831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.703312    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerDied","Data":"2a05031a00f8d0bc81c83538b58699c02b4be4dd2370fa19e1af0c9902bda857"}
Nov 02 14:15:02.703831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.703668    4492 scope.go:122] "RemoveContainer" containerID="2a05031a00f8d0bc81c83538b58699c02b4be4dd2370fa19e1af0c9902bda857"
Nov 02 14:15:02.705512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.705490    4492 generic.go:356] "Generic (PLEG): container finished" podID="13908eff-79d9-42d1-8a77-4d8649723420" containerID="265b6939e49e3f35aac6a32515a9893ddad5bf71a43c27d7f9127d160b0be1eb" exitCode=60
Nov 02 14:15:02.705613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.705581    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerDied","Data":"265b6939e49e3f35aac6a32515a9893ddad5bf71a43c27d7f9127d160b0be1eb"}
Nov 02 14:15:02.705898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.705875    4492 scope.go:122] "RemoveContainer" containerID="265b6939e49e3f35aac6a32515a9893ddad5bf71a43c27d7f9127d160b0be1eb"
Nov 02 14:15:02.708905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.708860    4492 generic.go:356] "Generic (PLEG): container finished" podID="a19418b4-2268-46ec-923d-de387729f6b7" containerID="27d21dfbefe7e4493f2296c03b35cf814bb3100cfad48ff0f229b483203a148b" exitCode=60
Nov 02 14:15:02.709043 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.709000    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerDied","Data":"27d21dfbefe7e4493f2296c03b35cf814bb3100cfad48ff0f229b483203a148b"}
Nov 02 14:15:02.709221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:02.709168    4492 scope.go:122] "RemoveContainer" containerID="27d21dfbefe7e4493f2296c03b35cf814bb3100cfad48ff0f229b483203a148b"
Nov 02 14:15:03.433587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.433551    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:03.716630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.716547    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:03.718606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.718569    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerStarted","Data":"bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9"}
Nov 02 14:15:03.720531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.720505    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerStarted","Data":"37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108"}
Nov 02 14:15:03.722447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.722424    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerStarted","Data":"ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424"}
Nov 02 14:15:03.723949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.723928    4492 generic.go:356] "Generic (PLEG): container finished" podID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerID="f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd" exitCode=0
Nov 02 14:15:03.724027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.723996    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" event={"ID":"0da14694-3a1f-4327-aa17-77ec3cd9549f","Type":"ContainerDied","Data":"f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"}
Nov 02 14:15:03.724314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.724301    4492 scope.go:122] "RemoveContainer" containerID="f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"
Nov 02 14:15:03.725675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.725644    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:15:03.725768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.725732    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:15:03.725838 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.725783    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:15:03.855908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:03.855781    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:04.469432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.463597    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"]
Nov 02 14:15:04.485422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.475733    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:15:04.615518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.615297    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:04.746894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.746798    4492 generic.go:356] "Generic (PLEG): container finished" podID="13908eff-79d9-42d1-8a77-4d8649723420" containerID="bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9" exitCode=60
Nov 02 14:15:04.747513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.747491    4492 scope.go:122] "RemoveContainer" containerID="bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9"
Nov 02 14:15:04.747734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.747708    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerDied","Data":"bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9"}
Nov 02 14:15:04.747880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.747866    4492 scope.go:122] "RemoveContainer" containerID="265b6939e49e3f35aac6a32515a9893ddad5bf71a43c27d7f9127d160b0be1eb"
Nov 02 14:15:04.748045 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:04.747869    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4_odh-ai-gateway-infra(13908eff-79d9-42d1-8a77-4d8649723420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" podUID="13908eff-79d9-42d1-8a77-4d8649723420"
Nov 02 14:15:04.764854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.764823    4492 generic.go:356] "Generic (PLEG): container finished" podID="a19418b4-2268-46ec-923d-de387729f6b7" containerID="37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108" exitCode=60
Nov 02 14:15:04.765006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.764861    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerDied","Data":"37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108"}
Nov 02 14:15:04.765208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.765194    4492 scope.go:122] "RemoveContainer" containerID="37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108"
Nov 02 14:15:04.765549 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:04.765518    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc_odh-ai-gateway-infra(a19418b4-2268-46ec-923d-de387729f6b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" podUID="a19418b4-2268-46ec-923d-de387729f6b7"
Nov 02 14:15:04.773396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.767465    4492 generic.go:356] "Generic (PLEG): container finished" podID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerID="ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424" exitCode=60
Nov 02 14:15:04.773396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.767540    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerDied","Data":"ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424"}
Nov 02 14:15:04.773396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.767928    4492 scope.go:122] "RemoveContainer" containerID="ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424"
Nov 02 14:15:04.773396 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:04.768444    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v_odh-ai-gateway-infra(5d74a3fa-03e9-497c-a3ab-8b3d011ad603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603"
Nov 02 14:15:04.773396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.770134    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" event={"ID":"0da14694-3a1f-4327-aa17-77ec3cd9549f","Type":"ContainerStarted","Data":"b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273"}
Nov 02 14:15:04.775386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.775371    4492 scope.go:122] "RemoveContainer" containerID="27d21dfbefe7e4493f2296c03b35cf814bb3100cfad48ff0f229b483203a148b"
Nov 02 14:15:04.789139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:04.789119    4492 scope.go:122] "RemoveContainer" containerID="2a05031a00f8d0bc81c83538b58699c02b4be4dd2370fa19e1af0c9902bda857"
Nov 02 14:15:05.394705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.394669    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:15:05.395475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.395455    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 14:15:05.395546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.395502    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 14:15:05.532406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.523922    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:05.787003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.786969    4492 scope.go:122] "RemoveContainer" containerID="ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424"
Nov 02 14:15:05.787514 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:05.787273    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v_odh-ai-gateway-infra(5d74a3fa-03e9-497c-a3ab-8b3d011ad603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603"
Nov 02 14:15:05.788686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.788667    4492 scope.go:122] "RemoveContainer" containerID="bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9"
Nov 02 14:15:05.788923 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:05.788905    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4_odh-ai-gateway-infra(13908eff-79d9-42d1-8a77-4d8649723420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" podUID="13908eff-79d9-42d1-8a77-4d8649723420"
Nov 02 14:15:05.790103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:05.790084    4492 scope.go:122] "RemoveContainer" containerID="37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108"
Nov 02 14:15:05.790268 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:05.790252    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc_odh-ai-gateway-infra(a19418b4-2268-46ec-923d-de387729f6b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" podUID="a19418b4-2268-46ec-923d-de387729f6b7"
Nov 02 14:15:06.205762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.205718    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:06.331137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.328969    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:06.395103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.394928    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 14:15:06.396823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.395804    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 14:15:06.675563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.675531    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"]
Nov 02 14:15:06.679094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.679064    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:15:06.860273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:06.860233    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:07.394835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:07.394790    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 14:15:07.395038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:07.394849    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 14:15:07.553337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:07.553304    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:08.394123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:08.394082    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 14:15:08.394587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:08.394148    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 14:15:09.327978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.327946    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:09.399406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.399286    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 14:15:09.399946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.399440    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 14:15:09.464021 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.463966    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm" podUID="cf067613-99dd-4278-aef5-a64464655c4c" containerName="payload-processing" containerID="cri-o://0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d" gracePeriod=25
Nov 02 14:15:09.808886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.808854    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:15:09.822785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.822754    4492 generic.go:356] "Generic (PLEG): container finished" podID="cf067613-99dd-4278-aef5-a64464655c4c" containerID="0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d" exitCode=0
Nov 02 14:15:09.822987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.822795    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"
Nov 02 14:15:09.822987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.822840    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm" event={"ID":"cf067613-99dd-4278-aef5-a64464655c4c","Type":"ContainerDied","Data":"0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d"}
Nov 02 14:15:09.822987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.822916    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm" event={"ID":"cf067613-99dd-4278-aef5-a64464655c4c","Type":"ContainerDied","Data":"8ce8834cf8dde1817e8c5692260cb13f85d8b715614096d2f64fa58ea3b84b16"}
Nov 02 14:15:09.822987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.822939    4492 scope.go:122] "RemoveContainer" containerID="0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d"
Nov 02 14:15:09.866393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.865552    4492 scope.go:122] "RemoveContainer" containerID="0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d"
Nov 02 14:15:09.871225 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:09.870961    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d\": container with ID starting with 0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d not found: ID does not exist" containerID="0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d"
Nov 02 14:15:09.871225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.871011    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d"} err="failed to get container status \"0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d\": rpc error: code = NotFound desc = could not find container \"0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d\": container with ID starting with 0b876c56744003ef6d083d876beb1f0ef321d551c7fc0d60c4e66a2a0d6a500d not found: ID does not exist"
Nov 02 14:15:09.920265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920229    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume\") pod \"cf067613-99dd-4278-aef5-a64464655c4c\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") "
Nov 02 14:15:09.920446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920334    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume\") pod \"cf067613-99dd-4278-aef5-a64464655c4c\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") "
Nov 02 14:15:09.920446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920400    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr\") pod \"cf067613-99dd-4278-aef5-a64464655c4c\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") "
Nov 02 14:15:09.920446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920437    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials\") pod \"cf067613-99dd-4278-aef5-a64464655c4c\" (UID: \"cf067613-99dd-4278-aef5-a64464655c4c\") "
Nov 02 14:15:09.920697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920671    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume" pod "cf067613-99dd-4278-aef5-a64464655c4c" (UID: "cf067613-99dd-4278-aef5-a64464655c4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:09.920766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920738    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume" pod "cf067613-99dd-4278-aef5-a64464655c4c" (UID: "cf067613-99dd-4278-aef5-a64464655c4c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:09.920891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920870    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:09.920952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.920893    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf067613-99dd-4278-aef5-a64464655c4c-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:09.922792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.922769    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr" pod "cf067613-99dd-4278-aef5-a64464655c4c" (UID: "cf067613-99dd-4278-aef5-a64464655c4c"). InnerVolumeSpecName "kube-api-access-7vvpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:09.922855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:09.922808    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials" pod "cf067613-99dd-4278-aef5-a64464655c4c" (UID: "cf067613-99dd-4278-aef5-a64464655c4c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:10.022004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.021968    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvpr\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-kube-api-access-7vvpr\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:10.022004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.022000    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf067613-99dd-4278-aef5-a64464655c4c-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:10.065248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.065171    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:10.192781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.192736    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"]
Nov 02 14:15:10.202602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.202570    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bf0b0d/payload-processing-external-model-e2e-extmod-a-bf0b0d-964dfd2dm"]
Nov 02 14:15:10.544307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.544267    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:15:10.545086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.545055    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:15:10.546417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.546393    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:15:10.831450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.831338    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:10.977019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:10.976898    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:11.273745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:11.273710    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:11.584016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:11.583933    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf067613-99dd-4278-aef5-a64464655c4c" path="/var/lib/kubelet/pods/cf067613-99dd-4278-aef5-a64464655c4c/volumes"
Nov 02 14:15:11.676833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:11.676791    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8" podUID="4a3d6957-6141-4a18-82e0-d5e83ceb918c" containerName="payload-processing" containerID="cri-o://19a887f44f022226f1c21d85ef10d027c44cefe391a18596f4980b0143172b0c" gracePeriod=25
Nov 02 14:15:11.706013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:11.705962    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:11.936396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:11.931867    4492 generic.go:356] "Generic (PLEG): container finished" podID="4a3d6957-6141-4a18-82e0-d5e83ceb918c" containerID="19a887f44f022226f1c21d85ef10d027c44cefe391a18596f4980b0143172b0c" exitCode=0
Nov 02 14:15:11.936396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:11.933184    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8" event={"ID":"4a3d6957-6141-4a18-82e0-d5e83ceb918c","Type":"ContainerDied","Data":"19a887f44f022226f1c21d85ef10d027c44cefe391a18596f4980b0143172b0c"}
Nov 02 14:15:12.117222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.117191    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:15:12.185059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185027    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials\") pod \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") "
Nov 02 14:15:12.185258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185079    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume\") pod \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") "
Nov 02 14:15:12.185258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185226    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5\") pod \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") "
Nov 02 14:15:12.185392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185274    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume\") pod \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\" (UID: \"4a3d6957-6141-4a18-82e0-d5e83ceb918c\") "
Nov 02 14:15:12.185519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185497    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume" pod "4a3d6957-6141-4a18-82e0-d5e83ceb918c" (UID: "4a3d6957-6141-4a18-82e0-d5e83ceb918c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:12.185776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185749    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume" pod "4a3d6957-6141-4a18-82e0-d5e83ceb918c" (UID: "4a3d6957-6141-4a18-82e0-d5e83ceb918c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:12.185920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.185789    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:12.187554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.187489    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5" pod "4a3d6957-6141-4a18-82e0-d5e83ceb918c" (UID: "4a3d6957-6141-4a18-82e0-d5e83ceb918c"). InnerVolumeSpecName "kube-api-access-bj4l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:12.187678 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.187659    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials" pod "4a3d6957-6141-4a18-82e0-d5e83ceb918c" (UID: "4a3d6957-6141-4a18-82e0-d5e83ceb918c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:12.287441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.287410    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj4l5\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-kube-api-access-bj4l5\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:12.287441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.287438    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3d6957-6141-4a18-82e0-d5e83ceb918c-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:12.287441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.287448    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a3d6957-6141-4a18-82e0-d5e83ceb918c-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:12.956124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.953138    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8" event={"ID":"4a3d6957-6141-4a18-82e0-d5e83ceb918c","Type":"ContainerDied","Data":"2a7bed3e58fa1a5abe06f457822122e62ebc8d73f1e2a11ca6bdec336783bf92"}
Nov 02 14:15:12.956124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.953216    4492 scope.go:122] "RemoveContainer" containerID="19a887f44f022226f1c21d85ef10d027c44cefe391a18596f4980b0143172b0c"
Nov 02 14:15:12.956124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:12.953330    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"
Nov 02 14:15:13.166964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:13.166930    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"]
Nov 02 14:15:13.202679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:13.202629    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-84f9t67m8"]
Nov 02 14:15:13.371456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:13.367960    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:13.600380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:13.600129    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3d6957-6141-4a18-82e0-d5e83ceb918c" path="/var/lib/kubelet/pods/4a3d6957-6141-4a18-82e0-d5e83ceb918c/volumes"
Nov 02 14:15:14.574111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:14.574078    4492 scope.go:122] "RemoveContainer" containerID="ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424"
Nov 02 14:15:14.978184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:14.978147    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerStarted","Data":"05180fcc4222470f23df2bce9205b501408e921f694d5ad2b0050fac419e418f"}
Nov 02 14:15:15.444698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.444655    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:15.588394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.588321    4492 scope.go:122] "RemoveContainer" containerID="bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9"
Nov 02 14:15:15.732130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.732042    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"]
Nov 02 14:15:15.732960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.732936    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4a3d6957-6141-4a18-82e0-d5e83ceb918c" containerName="payload-processing"
Nov 02 14:15:15.733184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.733168    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3d6957-6141-4a18-82e0-d5e83ceb918c" containerName="payload-processing"
Nov 02 14:15:15.733329 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.733301    4492 cpu_manager.go:414] "containerMap: removing container" podUID="cf067613-99dd-4278-aef5-a64464655c4c" containerName="payload-processing"
Nov 02 14:15:15.733443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.733429    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf067613-99dd-4278-aef5-a64464655c4c" containerName="payload-processing"
Nov 02 14:15:15.733759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.733742    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3d6957-6141-4a18-82e0-d5e83ceb918c" containerName="payload-processing"
Nov 02 14:15:15.733929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.733898    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf067613-99dd-4278-aef5-a64464655c4c" containerName="payload-processing"
Nov 02 14:15:15.739694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.739673    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.759069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.759042    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"openshift-service-ca.crt\""
Nov 02 14:15:15.759409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.759386    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"kube-root-ca.crt\""
Nov 02 14:15:15.767407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.765312    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"]
Nov 02 14:15:15.768963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.768914    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"default-dockercfg-9x58g\""
Nov 02 14:15:15.769116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.768933    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"simulator-w3-866159-kserve-self-signed-certs\""
Nov 02 14:15:15.785750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.785719    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.785750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.785757    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.785995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.785808    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.786108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.786083    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.786172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.786126    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztv4\" (UniqueName: \"kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.786260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.786244    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.812758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.812725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:15.887151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887093    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887153    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztv4\" (UniqueName: \"kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887216    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887263    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887293    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887404    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887749 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887721    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887807    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.887883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.887822    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.889964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.889934    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.890150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.890135    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:15.909401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:15.909373    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztv4\" (UniqueName: \"kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4\") pod \"simulator-w3-866159-kserve-9bb448ccc-ldxxg\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") " pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:16.018792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.017373    4492 generic.go:356] "Generic (PLEG): container finished" podID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerID="05180fcc4222470f23df2bce9205b501408e921f694d5ad2b0050fac419e418f" exitCode=60
Nov 02 14:15:16.018792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.017637    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerDied","Data":"05180fcc4222470f23df2bce9205b501408e921f694d5ad2b0050fac419e418f"}
Nov 02 14:15:16.018792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.017715    4492 scope.go:122] "RemoveContainer" containerID="ac30a958eb1a5db55ca7d5ab7e9ea14f491cda2826c0b32d65113e612dc6d424"
Nov 02 14:15:16.018792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.018257    4492 scope.go:122] "RemoveContainer" containerID="05180fcc4222470f23df2bce9205b501408e921f694d5ad2b0050fac419e418f"
Nov 02 14:15:16.018792 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:16.018679    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v_odh-ai-gateway-infra(5d74a3fa-03e9-497c-a3ab-8b3d011ad603)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603"
Nov 02 14:15:16.054324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.053287    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerStarted","Data":"8a9ef3fc65646d767c5cc3bec3f50c8b96aa80e5ea375382719b1de2ac431d7f"}
Nov 02 14:15:16.074524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.074468    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"]
Nov 02 14:15:16.093515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.074957    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:16.437164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.437122    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"]
Nov 02 14:15:16.439950 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:15:16.439883    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26d970fc_38ab_48a8_aab2_984e07c2b1d3.slice/crio-f694dbfeff2097b55251bda2501dff830685b1197e611c83b29d460f50bb3b9e WatchSource:0}: Error finding container f694dbfeff2097b55251bda2501dff830685b1197e611c83b29d460f50bb3b9e: Status 404 returned error can't find the container with id f694dbfeff2097b55251bda2501dff830685b1197e611c83b29d460f50bb3b9e
Nov 02 14:15:16.527738 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:16.527651    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:15:16.975571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:16.975532    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:17.066451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.066416    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" event={"ID":"26d970fc-38ab-48a8-aab2-984e07c2b1d3","Type":"ContainerStarted","Data":"054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9"}
Nov 02 14:15:17.066612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.066468    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" event={"ID":"26d970fc-38ab-48a8-aab2-984e07c2b1d3","Type":"ContainerStarted","Data":"f694dbfeff2097b55251bda2501dff830685b1197e611c83b29d460f50bb3b9e"}
Nov 02 14:15:17.068784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.068748    4492 generic.go:356] "Generic (PLEG): container finished" podID="13908eff-79d9-42d1-8a77-4d8649723420" containerID="8a9ef3fc65646d767c5cc3bec3f50c8b96aa80e5ea375382719b1de2ac431d7f" exitCode=60
Nov 02 14:15:17.068949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.068812    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerDied","Data":"8a9ef3fc65646d767c5cc3bec3f50c8b96aa80e5ea375382719b1de2ac431d7f"}
Nov 02 14:15:17.068949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.068875    4492 scope.go:122] "RemoveContainer" containerID="bc49c4222e96f688fea814f00410f8075aa1c14679406c96352a3605866249b9"
Nov 02 14:15:17.069522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.069504    4492 scope.go:122] "RemoveContainer" containerID="8a9ef3fc65646d767c5cc3bec3f50c8b96aa80e5ea375382719b1de2ac431d7f"
Nov 02 14:15:17.069857 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:17.069834    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4_odh-ai-gateway-infra(13908eff-79d9-42d1-8a77-4d8649723420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" podUID="13908eff-79d9-42d1-8a77-4d8649723420"
Nov 02 14:15:17.075117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.075092    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:17.154013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.153951    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"]
Nov 02 14:15:17.191509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.191461    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" podStartSLOduration=2.191446277 podStartE2EDuration="2.191446277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:17.186217684 +0000 UTC m=+1081.729933277" watchObservedRunningTime="2026-11-02 14:15:17.191446277 +0000 UTC m=+1081.735161870"
Nov 02 14:15:17.264934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.264909    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:17.410049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.409783    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca\") pod \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") "
Nov 02 14:15:17.410260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.410062    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55\" (UniqueName: \"kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55\") pod \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\" (UID: \"5d74a3fa-03e9-497c-a3ab-8b3d011ad603\") "
Nov 02 14:15:17.411019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.410961    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca" pod "5d74a3fa-03e9-497c-a3ab-8b3d011ad603" (UID: "5d74a3fa-03e9-497c-a3ab-8b3d011ad603"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:17.414013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.413984    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55" pod "5d74a3fa-03e9-497c-a3ab-8b3d011ad603" (UID: "5d74a3fa-03e9-497c-a3ab-8b3d011ad603"). InnerVolumeSpecName "kube-api-access-x6j55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:17.512134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.512048    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:17.512134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.512076    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6j55\" (UniqueName: \"kubernetes.io/projected/5d74a3fa-03e9-497c-a3ab-8b3d011ad603-kube-api-access-x6j55\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:17.583268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.583224    4492 scope.go:122] "RemoveContainer" containerID="37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108"
Nov 02 14:15:17.625475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:17.625444    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:18.077779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.077748    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"
Nov 02 14:15:18.078264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.077783    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v" event={"ID":"5d74a3fa-03e9-497c-a3ab-8b3d011ad603","Type":"ContainerDied","Data":"8ce6ac2d5a353701a89f56b9d3ff69a247d6143894455f131ff98be362101646"}
Nov 02 14:15:18.078264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.077841    4492 scope.go:122] "RemoveContainer" containerID="05180fcc4222470f23df2bce9205b501408e921f694d5ad2b0050fac419e418f"
Nov 02 14:15:18.082931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.082896    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerStarted","Data":"27b49f36c57f393951972a70d81dd8759ee82ed5c6ada111862b7c235e6cc754"}
Nov 02 14:15:18.168869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.168831    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"]
Nov 02 14:15:18.180255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.180217    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bf0b0d-29893815-w6p8v"]
Nov 02 14:15:18.250150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.250118    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:18.323103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.323067    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r\" (UniqueName: \"kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r\") pod \"13908eff-79d9-42d1-8a77-4d8649723420\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") "
Nov 02 14:15:18.323304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.323173    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca\") pod \"13908eff-79d9-42d1-8a77-4d8649723420\" (UID: \"13908eff-79d9-42d1-8a77-4d8649723420\") "
Nov 02 14:15:18.323647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.323617    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca" pod "13908eff-79d9-42d1-8a77-4d8649723420" (UID: "13908eff-79d9-42d1-8a77-4d8649723420"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:18.325467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.325443    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r" pod "13908eff-79d9-42d1-8a77-4d8649723420" (UID: "13908eff-79d9-42d1-8a77-4d8649723420"). InnerVolumeSpecName "kube-api-access-dzg6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:18.425140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.425104    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg6r\" (UniqueName: \"kubernetes.io/projected/13908eff-79d9-42d1-8a77-4d8649723420-kube-api-access-dzg6r\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:18.425140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:18.425134    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13908eff-79d9-42d1-8a77-4d8649723420-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.091318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.091226    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"
Nov 02 14:15:19.091318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.091277    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4" event={"ID":"13908eff-79d9-42d1-8a77-4d8649723420","Type":"ContainerDied","Data":"71bf9334703c62743ff3c753320778eb56f78fe7ac160ddfefe8b01b36ec519c"}
Nov 02 14:15:19.091849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.091345    4492 scope.go:122] "RemoveContainer" containerID="8a9ef3fc65646d767c5cc3bec3f50c8b96aa80e5ea375382719b1de2ac431d7f"
Nov 02 14:15:19.093492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.093469    4492 generic.go:356] "Generic (PLEG): container finished" podID="a19418b4-2268-46ec-923d-de387729f6b7" containerID="27b49f36c57f393951972a70d81dd8759ee82ed5c6ada111862b7c235e6cc754" exitCode=60
Nov 02 14:15:19.093613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.093546    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerDied","Data":"27b49f36c57f393951972a70d81dd8759ee82ed5c6ada111862b7c235e6cc754"}
Nov 02 14:15:19.093985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.093966    4492 scope.go:122] "RemoveContainer" containerID="27b49f36c57f393951972a70d81dd8759ee82ed5c6ada111862b7c235e6cc754"
Nov 02 14:15:19.094324 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:19.094301    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc_odh-ai-gateway-infra(a19418b4-2268-46ec-923d-de387729f6b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" podUID="a19418b4-2268-46ec-923d-de387729f6b7"
Nov 02 14:15:19.102200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.102181    4492 scope.go:122] "RemoveContainer" containerID="37219149fe24e38d7ad4404d1616d69d8e6d458f4b01ab33f0020f70a382b108"
Nov 02 14:15:19.138239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.138211    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"]
Nov 02 14:15:19.147340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.147309    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"]
Nov 02 14:15:19.157299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.157273    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-514524-29893815-qmfm4"]
Nov 02 14:15:19.583391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.583337    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13908eff-79d9-42d1-8a77-4d8649723420" path="/var/lib/kubelet/pods/13908eff-79d9-42d1-8a77-4d8649723420/volumes"
Nov 02 14:15:19.583803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.583787    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" path="/var/lib/kubelet/pods/5d74a3fa-03e9-497c-a3ab-8b3d011ad603/volumes"
Nov 02 14:15:19.601040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:19.600994    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:20.258390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.258345    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:20.352099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.352065    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca\") pod \"a19418b4-2268-46ec-923d-de387729f6b7\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") "
Nov 02 14:15:20.352299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.352113    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl\" (UniqueName: \"kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl\") pod \"a19418b4-2268-46ec-923d-de387729f6b7\" (UID: \"a19418b4-2268-46ec-923d-de387729f6b7\") "
Nov 02 14:15:20.352464 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.352441    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca" pod "a19418b4-2268-46ec-923d-de387729f6b7" (UID: "a19418b4-2268-46ec-923d-de387729f6b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:20.352680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.352664    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19418b4-2268-46ec-923d-de387729f6b7-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:20.354551 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.354524    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl" pod "a19418b4-2268-46ec-923d-de387729f6b7" (UID: "a19418b4-2268-46ec-923d-de387729f6b7"). InnerVolumeSpecName "kube-api-access-5vdjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:20.453704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:20.453673    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdjl\" (UniqueName: \"kubernetes.io/projected/a19418b4-2268-46ec-923d-de387729f6b7-kube-api-access-5vdjl\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:21.114851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.114815    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"
Nov 02 14:15:21.114851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.114823    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc" event={"ID":"a19418b4-2268-46ec-923d-de387729f6b7","Type":"ContainerDied","Data":"52f99a288baff1468b39bcd4010d39ff4059ed34284dc8c02e4c5ff951a7685e"}
Nov 02 14:15:21.115127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.114883    4492 scope.go:122] "RemoveContainer" containerID="27b49f36c57f393951972a70d81dd8759ee82ed5c6ada111862b7c235e6cc754"
Nov 02 14:15:21.162117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.162083    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"]
Nov 02 14:15:21.168251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.168226    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-de8cc7-29893815-4rpvc"]
Nov 02 14:15:21.559282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.559247    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:21.582077 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:21.582046    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19418b4-2268-46ec-923d-de387729f6b7" path="/var/lib/kubelet/pods/a19418b4-2268-46ec-923d-de387729f6b7/volumes"
Nov 02 14:15:23.539399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:23.539330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:25.421248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:25.421215    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:26.075301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:26.075245    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:26.089440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:26.089402    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:26.163601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:26.163552    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:15:27.346481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:27.346447    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:29.930780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:29.930735    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:30.077687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:30.077651    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:30.215435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:30.210984    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:30.491822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:30.491745    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:31.268797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:31.268766    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:31.434615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:31.434581    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:32.108677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:32.108641    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:32.254394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:32.254328    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:32.397427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:32.396720    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:33.795091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795052    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"]
Nov 02 14:15:33.795927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795905    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795930    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795945    4492 cpu_manager.go:414] "containerMap: removing container" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795950    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795960    4492 cpu_manager.go:414] "containerMap: removing container" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795965    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795981    4492 cpu_manager.go:414] "containerMap: removing container" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.795991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.795986    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796009    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796014    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796023    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796027    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796038    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796042    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796052    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796056    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796133    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796143    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796149    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796155    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d74a3fa-03e9-497c-a3ab-8b3d011ad603" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796161    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796168    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796174    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796178    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.796319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.796184    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="13908eff-79d9-42d1-8a77-4d8649723420" containerName="cleanup"
Nov 02 14:15:33.799978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.799957    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.815160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.815135    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"]
Nov 02 14:15:33.820793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.820765    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-1b7ad177ca79\""
Nov 02 14:15:33.882482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.874001    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.882482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.874075    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.882482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.874142    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42csp\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.882482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.874291    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.975968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.975936    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.976170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.976018    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.976170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.976063    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.976170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.976111    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42csp\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.976770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.976742    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.976866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.976828    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.978773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.978743    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:33.998464 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:33.998431    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42csp\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:34.114863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:34.114774    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:34.330214 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:34.330174    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:34.463404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:34.463377    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"]
Nov 02 14:15:34.465837 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:15:34.465804    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f3de4_971e_4f54_861f_d20f89bd631d.slice/crio-e77d96e5c666b052c0d9f440841b90cd0567f4aada3fdf1089aa5c30d3ba3b3b WatchSource:0}: Error finding container e77d96e5c666b052c0d9f440841b90cd0567f4aada3fdf1089aa5c30d3ba3b3b: Status 404 returned error can't find the container with id e77d96e5c666b052c0d9f440841b90cd0567f4aada3fdf1089aa5c30d3ba3b3b
Nov 02 14:15:35.410974 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:35.410937    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq" event={"ID":"c01f3de4-971e-4f54-861f-d20f89bd631d","Type":"ContainerStarted","Data":"6030ee0039892abac80d74fd4eae4591cc643b4532b9176fcf0edecd068e4aa4"}
Nov 02 14:15:35.411401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:35.410992    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq" event={"ID":"c01f3de4-971e-4f54-861f-d20f89bd631d","Type":"ContainerStarted","Data":"e77d96e5c666b052c0d9f440841b90cd0567f4aada3fdf1089aa5c30d3ba3b3b"}
Nov 02 14:15:35.411401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:35.411028    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:35.462013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:35.461961    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq" podStartSLOduration=2.461948553 podStartE2EDuration="2.461948553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:35.460602638 +0000 UTC m=+1100.004318296" watchObservedRunningTime="2026-11-02 14:15:35.461948553 +0000 UTC m=+1100.005664183"
Nov 02 14:15:36.157222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:36.157190    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:37.427650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.427623    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"]
Nov 02 14:15:37.428297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.428282    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:37.428297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.428298    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19418b4-2268-46ec-923d-de387729f6b7" containerName="cleanup"
Nov 02 14:15:37.432410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.432384    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.460848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.460817    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"]
Nov 02 14:15:37.460997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.460939    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"premium-w3-866159-kserve-self-signed-certs\""
Nov 02 14:15:37.485703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.485677    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:37.519580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.519549    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.519763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.519585    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.519763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.519615    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.519838 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.519756    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvkh\" (UniqueName: \"kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.519838 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.519790    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.519838 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.519808    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.630378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.629179    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.630378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.629245    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.630378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.629311    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.630378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.629427    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvkh\" (UniqueName: \"kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.630378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.629462    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.630378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.629486    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.631606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.631560    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.633255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.632076    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.649065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.649033    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.662239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.650063    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.662239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.650974    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.677159 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.677089    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvkh\" (UniqueName: \"kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh\") pod \"premium-w3-866159-kserve-7dccbdf864-wwb8b\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") " pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:37.751316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:37.751215    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:38.100577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:38.100283    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:38.115184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:38.115156    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"]
Nov 02 14:15:38.442182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:38.442143    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" event={"ID":"fea0c720-e5b0-4bd1-b7ee-4946166fed00","Type":"ContainerStarted","Data":"ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4"}
Nov 02 14:15:38.442182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:38.442187    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" event={"ID":"fea0c720-e5b0-4bd1-b7ee-4946166fed00","Type":"ContainerStarted","Data":"7ea5cb132fc5104ad16798402a899bba9015f569ee58b602db21af6913595cd6"}
Nov 02 14:15:38.483051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:38.482998    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" podStartSLOduration=1.482983934 podStartE2EDuration="1.482983934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:38.479388412 +0000 UTC m=+1103.023104003" watchObservedRunningTime="2026-11-02 14:15:38.482983934 +0000 UTC m=+1103.026699524"
Nov 02 14:15:38.751946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:38.751864    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:40.031387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:40.018922    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:41.948689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:41.948650    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:43.747700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:43.747657    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:45.632873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:45.632827    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:46.422045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:46.422010    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:15:46.520117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:46.520085    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"]
Nov 02 14:15:46.526592 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:46.526550    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:15:47.583966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:47.583936    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:47.751703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:47.751666    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:47.764735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:47.764703    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:48.409510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:48.409476    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:48.594349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:48.594321    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:15:49.436907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:49.436875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:51.227414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.227342    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:51.521093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.520999    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58" podUID="527c21f1-4cc3-45ef-8ec9-171d1b780acc" containerName="payload-processing" containerID="cri-o://b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce" gracePeriod=25
Nov 02 14:15:51.790988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.790926    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:15:51.853168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.853116    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials\") pod \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") "
Nov 02 14:15:51.853395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.853231    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume\") pod \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") "
Nov 02 14:15:51.853490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.853395    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh\") pod \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") "
Nov 02 14:15:51.853490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.853456    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume\") pod \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\" (UID: \"527c21f1-4cc3-45ef-8ec9-171d1b780acc\") "
Nov 02 14:15:51.854374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.854227    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume" pod "527c21f1-4cc3-45ef-8ec9-171d1b780acc" (UID: "527c21f1-4cc3-45ef-8ec9-171d1b780acc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:51.854737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.854698    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume" pod "527c21f1-4cc3-45ef-8ec9-171d1b780acc" (UID: "527c21f1-4cc3-45ef-8ec9-171d1b780acc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:51.867791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.867756    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh" pod "527c21f1-4cc3-45ef-8ec9-171d1b780acc" (UID: "527c21f1-4cc3-45ef-8ec9-171d1b780acc"). InnerVolumeSpecName "kube-api-access-824lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:51.867909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.867827    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials" pod "527c21f1-4cc3-45ef-8ec9-171d1b780acc" (UID: "527c21f1-4cc3-45ef-8ec9-171d1b780acc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:51.956406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.956345    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:51.956406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.956404    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824lh\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-kube-api-access-824lh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:51.956630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.956426    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527c21f1-4cc3-45ef-8ec9-171d1b780acc-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:51.956630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:51.956448    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527c21f1-4cc3-45ef-8ec9-171d1b780acc-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:15:52.610416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.610384    4492 generic.go:356] "Generic (PLEG): container finished" podID="527c21f1-4cc3-45ef-8ec9-171d1b780acc" containerID="b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce" exitCode=0
Nov 02 14:15:52.610416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.610416    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"
Nov 02 14:15:52.610921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.610461    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58" event={"ID":"527c21f1-4cc3-45ef-8ec9-171d1b780acc","Type":"ContainerDied","Data":"b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce"}
Nov 02 14:15:52.610921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.610518    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58" event={"ID":"527c21f1-4cc3-45ef-8ec9-171d1b780acc","Type":"ContainerDied","Data":"ed612614dadf61d47b0eba662743646a5531d764a0f2eabbe93529045cdeef8d"}
Nov 02 14:15:52.610921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.610541    4492 scope.go:122] "RemoveContainer" containerID="b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce"
Nov 02 14:15:52.628767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.628745    4492 scope.go:122] "RemoveContainer" containerID="b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce"
Nov 02 14:15:52.629069 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:15:52.629050    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce\": container with ID starting with b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce not found: ID does not exist" containerID="b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce"
Nov 02 14:15:52.629122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.629078    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce"} err="failed to get container status \"b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce\": rpc error: code = NotFound desc = could not find container \"b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce\": container with ID starting with b3f86d4f859e7888cfb6df6b30e05f3eed91c7a50ccf9c8f99a26a90a853cdce not found: ID does not exist"
Nov 02 14:15:52.662697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.662666    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"]
Nov 02 14:15:52.670480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:52.670453    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6ffff58"]
Nov 02 14:15:53.039736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:53.039693    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:53.582715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:53.582676    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527c21f1-4cc3-45ef-8ec9-171d1b780acc" path="/var/lib/kubelet/pods/527c21f1-4cc3-45ef-8ec9-171d1b780acc/volumes"
Nov 02 14:15:55.954572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:55.954535    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:56.759516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:56.759470    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" podUID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:15:58.403187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:58.403129    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:58.529427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:58.529376    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:15:59.327428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:15:59.327379    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:00.076692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.076655    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"]
Nov 02 14:16:00.077257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.077244    4492 cpu_manager.go:414] "containerMap: removing container" podUID="527c21f1-4cc3-45ef-8ec9-171d1b780acc" containerName="payload-processing"
Nov 02 14:16:00.077313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.077258    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527c21f1-4cc3-45ef-8ec9-171d1b780acc" containerName="payload-processing"
Nov 02 14:16:00.077421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.077410    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="527c21f1-4cc3-45ef-8ec9-171d1b780acc" containerName="payload-processing"
Nov 02 14:16:00.082272 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.082252    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.094779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.094756    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-3e3d57a25afd\""
Nov 02 14:16:00.108826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.108794    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"]
Nov 02 14:16:00.163106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.163071    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.163290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.163222    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.163290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.163276    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvs2\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.163470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.163456    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.264592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.264554    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.264773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.264633    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.264824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.264795    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvs2\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.265030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.265007    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.265462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.265438    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.265665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.265645    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.268195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.268166    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.286374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.286331    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvs2\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.397079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.396983    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:00.427897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.427839    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:00.749849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:00.749808    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"]
Nov 02 14:16:00.750494 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:16:00.750467    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3808fad_5f8c_4085_aae5_4cdc2be90f9b.slice/crio-35eb74b7819b3bb4c38f966bac7391b0f731212851a0ae4b947a68df4872b916 WatchSource:0}: Error finding container 35eb74b7819b3bb4c38f966bac7391b0f731212851a0ae4b947a68df4872b916: Status 404 returned error can't find the container with id 35eb74b7819b3bb4c38f966bac7391b0f731212851a0ae4b947a68df4872b916
Nov 02 14:16:01.143907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:01.143801    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v" event={"ID":"d3808fad-5f8c-4085-aae5-4cdc2be90f9b","Type":"ContainerStarted","Data":"df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb"}
Nov 02 14:16:01.143907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:01.143866    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v" event={"ID":"d3808fad-5f8c-4085-aae5-4cdc2be90f9b","Type":"ContainerStarted","Data":"35eb74b7819b3bb4c38f966bac7391b0f731212851a0ae4b947a68df4872b916"}
Nov 02 14:16:01.143907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:01.143903    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:01.183407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:01.183295    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v" podStartSLOduration=1.183275779 podStartE2EDuration="1.183275779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:01.181718982 +0000 UTC m=+1125.725434576" watchObservedRunningTime="2026-11-02 14:16:01.183275779 +0000 UTC m=+1125.726991375"
Nov 02 14:16:02.387257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:02.387221    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:04.385424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:04.385381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:04.514062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:04.514020    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:06.354698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:06.354668    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:06.972553 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:06.972522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:08.110135 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:08.110103    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:09.894645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:09.894604    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:11.743240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:11.743203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:11.864453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:11.864421    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:12.154946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:12.154857    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:12.240041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:12.239987    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"]
Nov 02 14:16:13.836427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:13.827597    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:16.235373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:16.235334    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:16.296596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:16.295633    4492 scope.go:122] "RemoveContainer" containerID="4e47828aea9c1fce7dd17e90a4b7a32d54e9dcf706e3fb9c3485b94e77e63c7d"
Nov 02 14:16:16.518267 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:16:16.518186    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:16:17.207376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.207325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:17.241669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.241627    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq" podUID="c01f3de4-971e-4f54-861f-d20f89bd631d" containerName="payload-processing" containerID="cri-o://6030ee0039892abac80d74fd4eae4591cc643b4532b9176fcf0edecd068e4aa4" gracePeriod=25
Nov 02 14:16:17.396517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.396476    4492 generic.go:356] "Generic (PLEG): container finished" podID="c01f3de4-971e-4f54-861f-d20f89bd631d" containerID="6030ee0039892abac80d74fd4eae4591cc643b4532b9176fcf0edecd068e4aa4" exitCode=0
Nov 02 14:16:17.396711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.396604    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq" event={"ID":"c01f3de4-971e-4f54-861f-d20f89bd631d","Type":"ContainerDied","Data":"6030ee0039892abac80d74fd4eae4591cc643b4532b9176fcf0edecd068e4aa4"}
Nov 02 14:16:17.515115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.515090    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:16:17.619921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.619884    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp\") pod \"c01f3de4-971e-4f54-861f-d20f89bd631d\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") "
Nov 02 14:16:17.620111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.619991    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume\") pod \"c01f3de4-971e-4f54-861f-d20f89bd631d\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") "
Nov 02 14:16:17.620111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.620023    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials\") pod \"c01f3de4-971e-4f54-861f-d20f89bd631d\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") "
Nov 02 14:16:17.620239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.620132    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume\") pod \"c01f3de4-971e-4f54-861f-d20f89bd631d\" (UID: \"c01f3de4-971e-4f54-861f-d20f89bd631d\") "
Nov 02 14:16:17.620522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.620489    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume" pod "c01f3de4-971e-4f54-861f-d20f89bd631d" (UID: "c01f3de4-971e-4f54-861f-d20f89bd631d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:16:17.620522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.620519    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume" pod "c01f3de4-971e-4f54-861f-d20f89bd631d" (UID: "c01f3de4-971e-4f54-861f-d20f89bd631d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:16:17.620694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.620650    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:17.620694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.620672    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c01f3de4-971e-4f54-861f-d20f89bd631d-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:17.622244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.622219    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp" pod "c01f3de4-971e-4f54-861f-d20f89bd631d" (UID: "c01f3de4-971e-4f54-861f-d20f89bd631d"). InnerVolumeSpecName "kube-api-access-42csp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:16:17.622365 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.622338    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials" pod "c01f3de4-971e-4f54-861f-d20f89bd631d" (UID: "c01f3de4-971e-4f54-861f-d20f89bd631d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:16:17.722268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.722223    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:17.722268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:17.722260    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42csp\" (UniqueName: \"kubernetes.io/projected/c01f3de4-971e-4f54-861f-d20f89bd631d-kube-api-access-42csp\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:18.027094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:18.027055    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:18.405372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:18.405258    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq" event={"ID":"c01f3de4-971e-4f54-861f-d20f89bd631d","Type":"ContainerDied","Data":"e77d96e5c666b052c0d9f440841b90cd0567f4aada3fdf1089aa5c30d3ba3b3b"}
Nov 02 14:16:18.405372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:18.405314    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"
Nov 02 14:16:18.405372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:18.405331    4492 scope.go:122] "RemoveContainer" containerID="6030ee0039892abac80d74fd4eae4591cc643b4532b9176fcf0edecd068e4aa4"
Nov 02 14:16:18.452759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:18.452727    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"]
Nov 02 14:16:18.462095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:18.462067    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79frdzq"]
Nov 02 14:16:19.581404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:19.581369    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01f3de4-971e-4f54-861f-d20f89bd631d" path="/var/lib/kubelet/pods/c01f3de4-971e-4f54-861f-d20f89bd631d/volumes"
Nov 02 14:16:19.905107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:19.905032    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:21.640179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.640133    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:21.709376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.705483    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"]
Nov 02 14:16:21.709376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.706662    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c01f3de4-971e-4f54-861f-d20f89bd631d" containerName="payload-processing"
Nov 02 14:16:21.709376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.706687    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01f3de4-971e-4f54-861f-d20f89bd631d" containerName="payload-processing"
Nov 02 14:16:21.709376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.706882    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01f3de4-971e-4f54-861f-d20f89bd631d" containerName="payload-processing"
Nov 02 14:16:21.736475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.728268    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.740851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.740803    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"]
Nov 02 14:16:21.745336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.745309    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2bb4ef-openshift-default-dockercfg-rlxt2\""
Nov 02 14:16:21.783430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783400    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783433    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783479    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783563    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783594    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783628    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783721    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d76sc\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.783917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783899    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.784023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.783966    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884709    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884767    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884789    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884832    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884856    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884887    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.884944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884933    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.884989    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d76sc\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.885110    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.885219    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.885298    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.885492    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.885560    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.885834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.885812    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.887217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.887190    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.887664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.887643    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.906730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.906667    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:21.906854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:21.906764    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d76sc\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc\") pod \"e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") " pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:22.056532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:22.056498    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:22.380612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:22.380586    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"]
Nov 02 14:16:22.384239 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:16:22.384205    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58065a4_78ba_4569_81f7_5f96483077d9.slice/crio-17f78a6059692559c39e454b37635d8647c1419734a93945c9df07ab42409b61 WatchSource:0}: Error finding container 17f78a6059692559c39e454b37635d8647c1419734a93945c9df07ab42409b61: Status 404 returned error can't find the container with id 17f78a6059692559c39e454b37635d8647c1419734a93945c9df07ab42409b61
Nov 02 14:16:22.386757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:22.386722    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:16:22.386856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:22.386788    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:16:22.386856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:22.386818    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:16:22.437766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:22.437743    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" event={"ID":"e58065a4-78ba-4569-81f7-5f96483077d9","Type":"ContainerStarted","Data":"17f78a6059692559c39e454b37635d8647c1419734a93945c9df07ab42409b61"}
Nov 02 14:16:23.427154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:23.427111    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:23.468404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:23.468085    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" event={"ID":"e58065a4-78ba-4569-81f7-5f96483077d9","Type":"ContainerStarted","Data":"86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c"}
Nov 02 14:16:24.058432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:24.057444    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:24.058779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:24.058749    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 14:16:24.058856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:24.058813    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 14:16:25.061379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:25.060801    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 14:16:25.061379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:25.060877    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 14:16:25.231932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:25.231901    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:26.065381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:26.057097    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 14:16:26.065381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:26.057208    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 14:16:26.957379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:26.955220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:27.064102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:27.057750    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 14:16:27.064102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:27.057827    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 14:16:28.060381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:28.058789    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 14:16:28.060381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:28.058857    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 14:16:28.852709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:28.852674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:29.062475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:29.062446    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:29.062964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:29.062847    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:29.063554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:29.063532    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:16:29.115940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:29.115833    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podStartSLOduration=8.115815723 podStartE2EDuration="8.115815723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:23.541698596 +0000 UTC m=+1148.085414194" watchObservedRunningTime="2026-11-02 14:16:29.115815723 +0000 UTC m=+1153.659531317"
Nov 02 14:16:32.526556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:32.526523    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:34.498739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:34.498702    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:36.292832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:36.292799    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:36.417406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:36.416189    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:37.254530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.254492    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"]
Nov 02 14:16:37.259012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.258992    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.262000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.261968    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.262141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.262024    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4skp\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.262202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.262158    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.262309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.262287    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.281383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.281323    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"]
Nov 02 14:16:37.363806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.363772    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.364285 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.363884    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.364285 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.363906    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.364285 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.363949    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4skp\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.364657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.364635    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.364705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.364662    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.367256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.367229    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.386039 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.386010    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4skp\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.574019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.573926    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:37.889059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:37.889029    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"]
Nov 02 14:16:37.891111 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:16:37.891084    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46f9bbc_b432_469e_b915_324b336d4a9d.slice/crio-baec51994bbbcf4438f3ae0147a0403a144fba76a07326635ac9381a3d51ee30 WatchSource:0}: Error finding container baec51994bbbcf4438f3ae0147a0403a144fba76a07326635ac9381a3d51ee30: Status 404 returned error can't find the container with id baec51994bbbcf4438f3ae0147a0403a144fba76a07326635ac9381a3d51ee30
Nov 02 14:16:38.231839 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.231807    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:38.475283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.475242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:38.678834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.678798    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2" event={"ID":"e46f9bbc-b432-469e-b915-324b336d4a9d","Type":"ContainerStarted","Data":"4251aa72dec296ab1f4051f855f27fe7ca55bd9857e3aaee1c98cd70ec7540c2"}
Nov 02 14:16:38.679049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.678858    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2" event={"ID":"e46f9bbc-b432-469e-b915-324b336d4a9d","Type":"ContainerStarted","Data":"baec51994bbbcf4438f3ae0147a0403a144fba76a07326635ac9381a3d51ee30"}
Nov 02 14:16:38.679049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.678927    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:38.718491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.718443    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2" podStartSLOduration=1.718403827 podStartE2EDuration="1.718403827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:38.717480422 +0000 UTC m=+1163.261196017" watchObservedRunningTime="2026-11-02 14:16:38.718403827 +0000 UTC m=+1163.262119421"
Nov 02 14:16:38.975574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.975497    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"]
Nov 02 14:16:38.980590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:38.980562    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.003393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.002723    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9ba664-openshift-default-dockercfg-h75gv\""
Nov 02 14:16:39.003393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.003386    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"]
Nov 02 14:16:39.032441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.032410    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:39.094186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094144    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094199    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094286    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094467    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094527    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094580    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094619    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094652    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72s9\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.094757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.094726    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.183118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.183040    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:39.202283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.202236    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.204109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.204081    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.204833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.204809    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.210606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.210562    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.211402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.211338    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72s9\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.216946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.216914    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.228498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.218701    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.229230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.229202    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.229058    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.217531    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.230138    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.211265    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.269816    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.340166    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72s9\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.350634    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.370732    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.370992    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.372388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.371316    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket\") pod \"e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") " pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:39.412393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.412347    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:39.540605 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.540514    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:39.604629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:39.604590    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:40.064347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.064313    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"]
Nov 02 14:16:40.064725 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:16:40.064699    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7177b655_6676_4a9f_8a9a_b9d4ab7f59c3.slice/crio-1cefa9da1c674498bd070feeed6328bfab5978bfb1b1421866ed7feca04f3644 WatchSource:0}: Error finding container 1cefa9da1c674498bd070feeed6328bfab5978bfb1b1421866ed7feca04f3644: Status 404 returned error can't find the container with id 1cefa9da1c674498bd070feeed6328bfab5978bfb1b1421866ed7feca04f3644
Nov 02 14:16:40.067559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.067519    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:16:40.067669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.067610    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:16:40.067669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.067656    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:16:40.181734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.181711    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:40.744548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.744149    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" event={"ID":"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3","Type":"ContainerStarted","Data":"961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"}
Nov 02 14:16:40.744548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.744216    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" event={"ID":"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3","Type":"ContainerStarted","Data":"1cefa9da1c674498bd070feeed6328bfab5978bfb1b1421866ed7feca04f3644"}
Nov 02 14:16:40.789566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:40.789386    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podStartSLOduration=2.789344614 podStartE2EDuration="2.789344614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:40.788143718 +0000 UTC m=+1165.331859324" watchObservedRunningTime="2026-11-02 14:16:40.789344614 +0000 UTC m=+1165.333060210"
Nov 02 14:16:41.605431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:41.605399    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:41.611302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:41.611273    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:41.754799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:41.754737    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:41.755718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:41.755695    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:16:42.000535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:42.000496    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"]
Nov 02 14:16:42.034647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:42.034613    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"]
Nov 02 14:16:42.164343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:42.164309    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:42.279093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:42.278999    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:42.405086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:42.405051    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:43.380857 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:43.380813    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:43.535579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:43.535544    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:43.911217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:43.911184    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:44.217890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:44.217855    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:45.097690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.097654    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:45.208672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.208642    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:45.271335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.271296    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"]
Nov 02 14:16:45.277201 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.277174    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.299884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.299851    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9ba664\""
Nov 02 14:16:45.299884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.299868    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9ba664-dockercfg-rbb8n\""
Nov 02 14:16:45.305271 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.304277    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"]
Nov 02 14:16:45.578225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.576713    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.578225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.577507    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69k6p\" (UniqueName: \"kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p\") pod \"payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.593468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.593437    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"]
Nov 02 14:16:45.603143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.603112    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"]
Nov 02 14:16:45.603407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.603277    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.679127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.679082    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.679338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.679159    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.679338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.679298    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69k6p\" (UniqueName: \"kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p\") pod \"payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.679546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.679340    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlfh\" (UniqueName: \"kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh\") pod \"payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.680040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.680019    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.701328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.701294    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69k6p\" (UniqueName: \"kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p\") pod \"payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.780917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.780883    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.781090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.780963    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlfh\" (UniqueName: \"kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh\") pod \"payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.781743 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.781722    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.802922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.802878    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlfh\" (UniqueName: \"kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh\") pod \"payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:45.892506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.892410    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:45.919969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:45.919930    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:46.118844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.118807    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:46.244175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.243175    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"]
Nov 02 14:16:46.276404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.275926    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"]
Nov 02 14:16:46.281642 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:16:46.281603    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231baf73_a109_4c55_ab96_306d004fa68f.slice/crio-a0c82176bf5c1cd47371e97c5a5c12e83bedab651a06d2ab88f6979d69e35536 WatchSource:0}: Error finding container a0c82176bf5c1cd47371e97c5a5c12e83bedab651a06d2ab88f6979d69e35536: Status 404 returned error can't find the container with id a0c82176bf5c1cd47371e97c5a5c12e83bedab651a06d2ab88f6979d69e35536
Nov 02 14:16:46.519459 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:16:46.519370    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:16:46.827650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.827548    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" event={"ID":"14ac141f-2e08-452f-a3d1-54bcf6484f68","Type":"ContainerStarted","Data":"978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6"}
Nov 02 14:16:46.827650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.827625    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:46.827650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.827648    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" event={"ID":"14ac141f-2e08-452f-a3d1-54bcf6484f68","Type":"ContainerStarted","Data":"7eb01d124663efaaa91ed992e6c9d0812a203dc030626b23d55c8c41ad551fa4"}
Nov 02 14:16:46.829383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.829339    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" event={"ID":"231baf73-a109-4c55-ab96-306d004fa68f","Type":"ContainerStarted","Data":"68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b"}
Nov 02 14:16:46.829516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.829410    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" event={"ID":"231baf73-a109-4c55-ab96-306d004fa68f","Type":"ContainerStarted","Data":"a0c82176bf5c1cd47371e97c5a5c12e83bedab651a06d2ab88f6979d69e35536"}
Nov 02 14:16:46.866956 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:46.866890    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" podStartSLOduration=1.866869754 podStartE2EDuration="1.866869754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:46.864939404 +0000 UTC m=+1171.408654999" watchObservedRunningTime="2026-11-02 14:16:46.866869754 +0000 UTC m=+1171.410585350"
Nov 02 14:16:47.001728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.001690    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt" podUID="59765dea-71c1-448c-bc17-cbde7df8b505" containerName="payload-processing" containerID="cri-o://5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd" gracePeriod=25
Nov 02 14:16:47.048386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.037481    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerName="payload-pre-processing" containerID="cri-o://f88ec4ecd7de1b8e62de5864c5a9c0e6e42ce18f0f14f6d4aec8774a09ee09ef" gracePeriod=25
Nov 02 14:16:47.313750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.313723    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:16:47.352652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.352595    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" podStartSLOduration=2.352573878 podStartE2EDuration="2.352573878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:46.901848883 +0000 UTC m=+1171.445564477" watchObservedRunningTime="2026-11-02 14:16:47.352573878 +0000 UTC m=+1171.896289477"
Nov 02 14:16:47.403827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.403742    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume\") pod \"59765dea-71c1-448c-bc17-cbde7df8b505\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") "
Nov 02 14:16:47.404006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.403993    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k\" (UniqueName: \"kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k\") pod \"59765dea-71c1-448c-bc17-cbde7df8b505\" (UID: \"59765dea-71c1-448c-bc17-cbde7df8b505\") "
Nov 02 14:16:47.404107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.404084    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume" pod "59765dea-71c1-448c-bc17-cbde7df8b505" (UID: "59765dea-71c1-448c-bc17-cbde7df8b505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:16:47.404300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.404288    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59765dea-71c1-448c-bc17-cbde7df8b505-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:47.406294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.406271    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k" pod "59765dea-71c1-448c-bc17-cbde7df8b505" (UID: "59765dea-71c1-448c-bc17-cbde7df8b505"). InnerVolumeSpecName "kube-api-access-4sb6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:16:47.505956 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.505919    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sb6k\" (UniqueName: \"kubernetes.io/projected/59765dea-71c1-448c-bc17-cbde7df8b505-kube-api-access-4sb6k\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:47.841468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.841428    4492 generic.go:356] "Generic (PLEG): container finished" podID="59765dea-71c1-448c-bc17-cbde7df8b505" containerID="5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd" exitCode=0
Nov 02 14:16:47.841672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.841471    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt" event={"ID":"59765dea-71c1-448c-bc17-cbde7df8b505","Type":"ContainerDied","Data":"5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd"}
Nov 02 14:16:47.841672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.841543    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt" event={"ID":"59765dea-71c1-448c-bc17-cbde7df8b505","Type":"ContainerDied","Data":"f8e5de303ed75c2f5d38bea92d7ea3f15e29a6590e8e77b20f88d43dec75c12e"}
Nov 02 14:16:47.841672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.841574    4492 scope.go:122] "RemoveContainer" containerID="5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd"
Nov 02 14:16:47.841856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.841839    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:47.841981 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.841962    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"
Nov 02 14:16:47.855044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.855022    4492 scope.go:122] "RemoveContainer" containerID="5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd"
Nov 02 14:16:47.855387 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:16:47.855366    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd\": container with ID starting with 5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd not found: ID does not exist" containerID="5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd"
Nov 02 14:16:47.855486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.855399    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd"} err="failed to get container status \"5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd\": rpc error: code = NotFound desc = could not find container \"5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd\": container with ID starting with 5afa60be1ddd39d9e88c7937bcddbd0cd351cc3b23810a910f6c92e9b3c088fd not found: ID does not exist"
Nov 02 14:16:47.893949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.893904    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"]
Nov 02 14:16:47.906099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.906051    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-de8cc7-d8b8475f9-227pt"]
Nov 02 14:16:47.911479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:47.911453    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:48.037919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:48.037872    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:48.158475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:48.158392    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:49.634380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:49.631988    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59765dea-71c1-448c-bc17-cbde7df8b505" path="/var/lib/kubelet/pods/59765dea-71c1-448c-bc17-cbde7df8b505/volumes"
Nov 02 14:16:49.754380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:49.753566    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:16:50.072380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:50.072227    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:50.217049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:50.149874    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"]
Nov 02 14:16:50.643851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:50.643822    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:51.099753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:51.099725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:52.451635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:52.451604    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:54.238084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:54.238050    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:55.150251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.150209    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v" podUID="d3808fad-5f8c-4085-aae5-4cdc2be90f9b" containerName="payload-processing" containerID="cri-o://df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb" gracePeriod=25
Nov 02 14:16:55.228627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.228586    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:16:55.421052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.421027    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:55.477609 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.477547    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2\") pod \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") "
Nov 02 14:16:55.477778 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.477741    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume\") pod \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") "
Nov 02 14:16:55.477893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.477874    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume\") pod \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") "
Nov 02 14:16:55.477981 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.477964    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials\") pod \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\" (UID: \"d3808fad-5f8c-4085-aae5-4cdc2be90f9b\") "
Nov 02 14:16:55.478234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.478207    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume" pod "d3808fad-5f8c-4085-aae5-4cdc2be90f9b" (UID: "d3808fad-5f8c-4085-aae5-4cdc2be90f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:16:55.478234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.478213    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume" pod "d3808fad-5f8c-4085-aae5-4cdc2be90f9b" (UID: "d3808fad-5f8c-4085-aae5-4cdc2be90f9b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:16:55.478569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.478514    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:55.478569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.478540    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:55.480398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.480342    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2" pod "d3808fad-5f8c-4085-aae5-4cdc2be90f9b" (UID: "d3808fad-5f8c-4085-aae5-4cdc2be90f9b"). InnerVolumeSpecName "kube-api-access-jnvs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:16:55.480797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.480773    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials" pod "d3808fad-5f8c-4085-aae5-4cdc2be90f9b" (UID: "d3808fad-5f8c-4085-aae5-4cdc2be90f9b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:16:55.587636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.580646    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnvs2\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-kube-api-access-jnvs2\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:55.587636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:55.580681    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3808fad-5f8c-4085-aae5-4cdc2be90f9b-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:16:56.037075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.037029    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:56.162387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.161783    4492 generic.go:356] "Generic (PLEG): container finished" podID="d3808fad-5f8c-4085-aae5-4cdc2be90f9b" containerID="df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb" exitCode=0
Nov 02 14:16:56.162387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.162001    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v" event={"ID":"d3808fad-5f8c-4085-aae5-4cdc2be90f9b","Type":"ContainerDied","Data":"df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb"}
Nov 02 14:16:56.162387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.162059    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v" event={"ID":"d3808fad-5f8c-4085-aae5-4cdc2be90f9b","Type":"ContainerDied","Data":"35eb74b7819b3bb4c38f966bac7391b0f731212851a0ae4b947a68df4872b916"}
Nov 02 14:16:56.162387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.162091    4492 scope.go:122] "RemoveContainer" containerID="df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb"
Nov 02 14:16:56.162387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.162261    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"
Nov 02 14:16:56.175327 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.175304    4492 scope.go:122] "RemoveContainer" containerID="df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb"
Nov 02 14:16:56.175699 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:16:56.175672    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb\": container with ID starting with df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb not found: ID does not exist" containerID="df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb"
Nov 02 14:16:56.175753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.175709    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb"} err="failed to get container status \"df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb\": rpc error: code = NotFound desc = could not find container \"df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb\": container with ID starting with df8401fe6be1c9b85a8f56b82bea8b2332c1175d985b773536511153c8b7f4eb not found: ID does not exist"
Nov 02 14:16:56.216472 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.216439    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"]
Nov 02 14:16:56.227322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.226606    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-7b6fgnn5v"]
Nov 02 14:16:56.867634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:56.867602    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:57.021071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.021036    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:57.144593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.144499    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:57.585335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.585294    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3808fad-5f8c-4085-aae5-4cdc2be90f9b" path="/var/lib/kubelet/pods/d3808fad-5f8c-4085-aae5-4cdc2be90f9b/volumes"
Nov 02 14:16:57.845461 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.845378    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:16:57.871372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.871332    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"]
Nov 02 14:16:57.871987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.871974    4492 cpu_manager.go:414] "containerMap: removing container" podUID="d3808fad-5f8c-4085-aae5-4cdc2be90f9b" containerName="payload-processing"
Nov 02 14:16:57.872066 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.871988    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3808fad-5f8c-4085-aae5-4cdc2be90f9b" containerName="payload-processing"
Nov 02 14:16:57.872066 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.872019    4492 cpu_manager.go:414] "containerMap: removing container" podUID="59765dea-71c1-448c-bc17-cbde7df8b505" containerName="payload-processing"
Nov 02 14:16:57.872066 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.872024    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59765dea-71c1-448c-bc17-cbde7df8b505" containerName="payload-processing"
Nov 02 14:16:57.872183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.872112    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3808fad-5f8c-4085-aae5-4cdc2be90f9b" containerName="payload-processing"
Nov 02 14:16:57.872183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.872126    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="59765dea-71c1-448c-bc17-cbde7df8b505" containerName="payload-processing"
Nov 02 14:16:57.877068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.877049    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:57.893391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.893221    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"]
Nov 02 14:16:57.911153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.911125    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpws9\" (UniqueName: \"kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:57.911345 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.911164    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:57.911426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.911377    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:57.911426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.911414    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:57.913067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.913047    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9ba664-serving-cert\""
Nov 02 14:16:57.969604 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:57.969575    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:58.012482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.012448    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.012679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.012570    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.012679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.012595    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.012802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.012679    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpws9\" (UniqueName: \"kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.012802 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:16:58.012740    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-serving-cert: secret "maas-api-e2e-extmod-a-9ba664-serving-cert" not found
Nov 02 14:16:58.012966 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:16:58.012812    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls podName:bfcc477a-aeee-4ff9-b64e-e8e10f501974 nodeName:}" failed. No retries permitted until 2026-11-02 14:16:58.512790657 +0000 UTC m=+1183.056506245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls") pod "maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" (UID: "bfcc477a-aeee-4ff9-b64e-e8e10f501974") : secret "maas-api-e2e-extmod-a-9ba664-serving-cert" not found
Nov 02 14:16:58.013274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.013251    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.015238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.015216    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.036146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.036114    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpws9\" (UniqueName: \"kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.103437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.096977    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:58.297630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.297593    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:58.521174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.521135    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.523734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.523709    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.668686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.668646    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:16:58.794784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.794461    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:16:58.946526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:58.946495    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:16:59.104242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:59.104216    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"]
Nov 02 14:16:59.106477 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:16:59.106445    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcc477a_aeee_4ff9_b64e_e8e10f501974.slice/crio-3ced21e3ade0b77fd6f7b28a0acb57a00fed4ae16d756da71c6dc8a303c8ef6b WatchSource:0}: Error finding container 3ced21e3ade0b77fd6f7b28a0acb57a00fed4ae16d756da71c6dc8a303c8ef6b: Status 404 returned error can't find the container with id 3ced21e3ade0b77fd6f7b28a0acb57a00fed4ae16d756da71c6dc8a303c8ef6b
Nov 02 14:16:59.289146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:59.289055    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" event={"ID":"bfcc477a-aeee-4ff9-b64e-e8e10f501974","Type":"ContainerStarted","Data":"3ced21e3ade0b77fd6f7b28a0acb57a00fed4ae16d756da71c6dc8a303c8ef6b"}
Nov 02 14:16:59.999824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:16:59.999793    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:00.299425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:00.299298    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" event={"ID":"bfcc477a-aeee-4ff9-b64e-e8e10f501974","Type":"ContainerStarted","Data":"a60d932e03ace640ecf710dd5d27ad02b9bc412b50448ea6cb0b137d93c74ecc"}
Nov 02 14:17:00.299595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:00.299431    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:17:00.340121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:00.340065    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" podStartSLOduration=2.803064342 podStartE2EDuration="3.340049626s" totalImagesPullingTime="536.985284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:00.337954077 +0000 UTC m=+1184.881669671" watchObservedRunningTime="2026-11-02 14:17:00.340049626 +0000 UTC m=+1184.883765268"
Nov 02 14:17:01.806890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:01.806858    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:02.241186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.241094    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:02.406912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.403398    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:02.687729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.687697    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"]
Nov 02 14:17:02.691940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.691920    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.712915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.712887    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"]
Nov 02 14:17:02.746915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.746873    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zhs\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.747198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.747174    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.747593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.747569    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.747769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.747738    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.849675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.849635    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zhs\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.850180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.849692    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.850180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.849918    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.850180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.849954    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.850618 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.850592    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.850829 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.850808    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.852741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.852717    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:02.874333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:02.874302    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zhs\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:03.006178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:03.006084    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:03.321255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:03.321226    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"]
Nov 02 14:17:03.324100 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:03.324068    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e8f414_bec1_4a11_acef_eac8d19d8b35.slice/crio-a24fc5c6d2c6454b91d4347bd33ff9493d051cedadeaac06420a0e0a31f9d1b3 WatchSource:0}: Error finding container a24fc5c6d2c6454b91d4347bd33ff9493d051cedadeaac06420a0e0a31f9d1b3: Status 404 returned error can't find the container with id a24fc5c6d2c6454b91d4347bd33ff9493d051cedadeaac06420a0e0a31f9d1b3
Nov 02 14:17:03.638725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:03.638640    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:03.747577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:03.747542    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:04.338595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:04.338558    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k" event={"ID":"04e8f414-bec1-4a11-acef-eac8d19d8b35","Type":"ContainerStarted","Data":"58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89"}
Nov 02 14:17:04.339230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:04.338619    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k" event={"ID":"04e8f414-bec1-4a11-acef-eac8d19d8b35","Type":"ContainerStarted","Data":"a24fc5c6d2c6454b91d4347bd33ff9493d051cedadeaac06420a0e0a31f9d1b3"}
Nov 02 14:17:04.339230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:04.338675    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:04.380124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:04.380070    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k" podStartSLOduration=2.380055214 podStartE2EDuration="2.380055214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:04.375979286 +0000 UTC m=+1188.919694876" watchObservedRunningTime="2026-11-02 14:17:04.380055214 +0000 UTC m=+1188.923770827"
Nov 02 14:17:05.227517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:05.227476    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:17:05.429165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:05.429123    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:06.312823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:06.312794    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:17:07.110864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.110834    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:07.234863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.234816    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:07.395191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.395153    4492 generic.go:356] "Generic (PLEG): container finished" podID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerID="f88ec4ecd7de1b8e62de5864c5a9c0e6e42ce18f0f14f6d4aec8774a09ee09ef" exitCode=0
Nov 02 14:17:07.395569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.395449    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" event={"ID":"c76db8eb-56c5-4d19-a55b-63dedde658c7","Type":"ContainerDied","Data":"f88ec4ecd7de1b8e62de5864c5a9c0e6e42ce18f0f14f6d4aec8774a09ee09ef"}
Nov 02 14:17:07.500768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.500318    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:17:07.542057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.540847    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs\" (UniqueName: \"kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs\") pod \"c76db8eb-56c5-4d19-a55b-63dedde658c7\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") "
Nov 02 14:17:07.542057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.540966    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume\") pod \"c76db8eb-56c5-4d19-a55b-63dedde658c7\" (UID: \"c76db8eb-56c5-4d19-a55b-63dedde658c7\") "
Nov 02 14:17:07.543756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.543692    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume" pod "c76db8eb-56c5-4d19-a55b-63dedde658c7" (UID: "c76db8eb-56c5-4d19-a55b-63dedde658c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:07.646436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.623587    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs" pod "c76db8eb-56c5-4d19-a55b-63dedde658c7" (UID: "c76db8eb-56c5-4d19-a55b-63dedde658c7"). InnerVolumeSpecName "kube-api-access-ntfqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:07.646436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.645037    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76db8eb-56c5-4d19-a55b-63dedde658c7-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:07.646436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:07.645061    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfqs\" (UniqueName: \"kubernetes.io/projected/c76db8eb-56c5-4d19-a55b-63dedde658c7-kube-api-access-ntfqs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:08.412376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:08.411636    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb" event={"ID":"c76db8eb-56c5-4d19-a55b-63dedde658c7","Type":"ContainerDied","Data":"bb53b0e56139844beb57f45986c82d0fafb782c0eeb418c68b5154060f833076"}
Nov 02 14:17:08.412376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:08.411726    4492 scope.go:122] "RemoveContainer" containerID="f88ec4ecd7de1b8e62de5864c5a9c0e6e42ce18f0f14f6d4aec8774a09ee09ef"
Nov 02 14:17:08.412943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:08.412418    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"
Nov 02 14:17:08.474744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:08.474611    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"]
Nov 02 14:17:08.491413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:08.491248    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-de8cc7-57d6dfbcbc-c4jsb"]
Nov 02 14:17:08.971677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:08.971639    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:09.660801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:09.659474    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" path="/var/lib/kubelet/pods/c76db8eb-56c5-4d19-a55b-63dedde658c7/volumes"
Nov 02 14:17:09.938939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:09.938906    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:10.446299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:10.446259    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:10.567572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:10.567532    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:10.694304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:10.694278    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:10.818410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:10.818319    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:10.977552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:10.977491    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:11.102174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.102076    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:11.228380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.224848    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:11.332645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.332604    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"]
Nov 02 14:17:11.334121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.333871    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerName="payload-pre-processing"
Nov 02 14:17:11.334399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.334382    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerName="payload-pre-processing"
Nov 02 14:17:11.334743 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.334728    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76db8eb-56c5-4d19-a55b-63dedde658c7" containerName="payload-pre-processing"
Nov 02 14:17:11.346734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.346708    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.370692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.370603    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"]
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416210    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416268    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416313    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416375    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416501    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416531    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416643    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416712    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.424400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.416773    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmvx\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518437    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518493    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518529    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmvx\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518573    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518596    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518620    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518646    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518700    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.518941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.518716    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.519480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.519452    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.519669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.519489    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.519669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.519509    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.519884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.519866    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.520285 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.520213    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.521064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.521043    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.521925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.521900    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.572097 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.572058    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.572854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.572833    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmvx\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.700210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.700178    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:11.796761 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.796669    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:11.911899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:11.911819    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:12.105382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.105207    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:12.167376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.167307    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"]
Nov 02 14:17:12.205153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.203732    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:12.205153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.203822    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:12.205153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.203870    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:12.517730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.517692    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" event={"ID":"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c","Type":"ContainerStarted","Data":"9a4c2262132ccea878f41683fe1a8a8880be688403ffce8cd180c46de0012947"}
Nov 02 14:17:12.518013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.517992    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" event={"ID":"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c","Type":"ContainerStarted","Data":"2c5feeced79084c3c42485a251869e1b6a4a905a431d8d24988a19fb13228dd5"}
Nov 02 14:17:12.704060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:12.703563    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:13.328704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.328648    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podStartSLOduration=2.32863234 podStartE2EDuration="2.32863234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:12.578256172 +0000 UTC m=+1197.121971767" watchObservedRunningTime="2026-11-02 14:17:13.32863234 +0000 UTC m=+1197.872347933"
Nov 02 14:17:13.328931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.328916    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"]
Nov 02 14:17:13.331920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.331899    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:13.378951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.378918    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"]
Nov 02 14:17:13.384633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.384607    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.399840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.399810    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"]
Nov 02 14:17:13.421004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.420003    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-4c75068f30fe\""
Nov 02 14:17:13.482339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.482302    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x79n\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.482569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.482412    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.482569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.482447    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.482569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.482471    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.602609 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.602511    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x79n\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.602609 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.602605    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.602844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.602639    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.602844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.602667    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.603574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.603548    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.606033 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.604684    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.614510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.614476    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.627006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.626973    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x79n\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.702744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.702702    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:13.702953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.702754    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:13.732007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.731971    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:13.811047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.810042    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:13.922559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:13.922474    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:14.412498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:14.412471    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"]
Nov 02 14:17:14.620891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:14.620846    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b" event={"ID":"76ba2e01-bfbc-48de-8098-f8cdc2638257","Type":"ContainerStarted","Data":"d16e0bfa486001d267779b134dcbf309c9316d4dc855e1c814671ce46637e505"}
Nov 02 14:17:14.713379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:14.713274    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:14.713595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:14.713446    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:15.018801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.018687    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:15.153382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.146976    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:15.629411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.629376    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b" event={"ID":"76ba2e01-bfbc-48de-8098-f8cdc2638257","Type":"ContainerStarted","Data":"5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2"}
Nov 02 14:17:15.629610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.629588    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:15.675483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.675428    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b" podStartSLOduration=2.675408456 podStartE2EDuration="2.675408456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:15.674653843 +0000 UTC m=+1200.218369438" watchObservedRunningTime="2026-11-02 14:17:15.675408456 +0000 UTC m=+1200.219124051"
Nov 02 14:17:15.688662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.688628    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:15.700768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.700731    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:15.700928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.700799    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:15.825657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:15.825623    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:16.075969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:16.075881    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:16.452380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:16.451255    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:16.598939 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:17:16.598906    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:17:16.701173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:16.701134    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:16.701405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:16.701185    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:16.869248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:16.868584    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:17.269507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.266085    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:17.388803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.388756    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:17.466378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.465730    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"]
Nov 02 14:17:17.473375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.473329    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.477531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.477503    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"]
Nov 02 14:17:17.497996 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.497958    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"]
Nov 02 14:17:17.501374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.500674    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f1e1c2-dockercfg-nrp7m\""
Nov 02 14:17:17.501374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.500738    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f1e1c2\""
Nov 02 14:17:17.523470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.519770    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:17.535513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.535443    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"]
Nov 02 14:17:17.673555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.673519    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vngh\" (UniqueName: \"kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.679284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.679247    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:17.683867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.683832    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6cx\" (UniqueName: \"kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx\") pod \"payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:17.684547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.684518    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.722091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.722042    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:17.800463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.726521    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:17.813380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.808235    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.813380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.808311    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vngh\" (UniqueName: \"kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.813380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.808457    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:17.813380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.808547    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6cx\" (UniqueName: \"kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx\") pod \"payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:17.813380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.809966    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.813782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.813523    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:17.875891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.875735    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vngh\" (UniqueName: \"kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:17.878363 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:17.878321    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6cx\" (UniqueName: \"kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx\") pod \"payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:18.111418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.109944    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:18.195562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.194149    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:18.329766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.329706    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k" podUID="04e8f414-bec1-4a11-acef-eac8d19d8b35" containerName="payload-processing" containerID="cri-o://58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89" gracePeriod=25
Nov 02 14:17:18.669719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.668733    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"]
Nov 02 14:17:18.676092 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:18.675345    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0ef4bd_8b1c_431f_a050_6e6d7ae2d93b.slice/crio-9d63ddf6a6d2aee01527f954ff19def49b6e464b5043b18d44584ff2537d9dee WatchSource:0}: Error finding container 9d63ddf6a6d2aee01527f954ff19def49b6e464b5043b18d44584ff2537d9dee: Status 404 returned error can't find the container with id 9d63ddf6a6d2aee01527f954ff19def49b6e464b5043b18d44584ff2537d9dee
Nov 02 14:17:18.677209 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.677090    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"]
Nov 02 14:17:18.692922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.692888    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:18.701893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.701866    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:18.701975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.701937    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:18.721016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.720914    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume\") pod \"04e8f414-bec1-4a11-acef-eac8d19d8b35\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") "
Nov 02 14:17:18.721121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721067    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume\") pod \"04e8f414-bec1-4a11-acef-eac8d19d8b35\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") "
Nov 02 14:17:18.721230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721204    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials\") pod \"04e8f414-bec1-4a11-acef-eac8d19d8b35\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") "
Nov 02 14:17:18.721389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721238    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs\") pod \"04e8f414-bec1-4a11-acef-eac8d19d8b35\" (UID: \"04e8f414-bec1-4a11-acef-eac8d19d8b35\") "
Nov 02 14:17:18.721621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721584    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume" pod "04e8f414-bec1-4a11-acef-eac8d19d8b35" (UID: "04e8f414-bec1-4a11-acef-eac8d19d8b35"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:18.721694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721645    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume" pod "04e8f414-bec1-4a11-acef-eac8d19d8b35" (UID: "04e8f414-bec1-4a11-acef-eac8d19d8b35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:18.721817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721796    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:18.721899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.721822    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e8f414-bec1-4a11-acef-eac8d19d8b35-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:18.724816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.724775    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials" pod "04e8f414-bec1-4a11-acef-eac8d19d8b35" (UID: "04e8f414-bec1-4a11-acef-eac8d19d8b35"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:18.724921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.724813    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs" pod "04e8f414-bec1-4a11-acef-eac8d19d8b35" (UID: "04e8f414-bec1-4a11-acef-eac8d19d8b35"). InnerVolumeSpecName "kube-api-access-x5zhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:18.765386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.761740    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:18.823063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.823025    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:18.823063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.823058    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zhs\" (UniqueName: \"kubernetes.io/projected/04e8f414-bec1-4a11-acef-eac8d19d8b35-kube-api-access-x5zhs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:18.826703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.826037    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" event={"ID":"7181ff04-9a1e-40c8-b261-81ffcc373eef","Type":"ContainerStarted","Data":"fcb5fb841c22375c53d31f83c934163386267dd181aabb16a496989cd33cddc9"}
Nov 02 14:17:18.826703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.826102    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" event={"ID":"7181ff04-9a1e-40c8-b261-81ffcc373eef","Type":"ContainerStarted","Data":"81b2606cde8567219bb7208d69d9c5d2ff982af8698b42174391b877cdcc2b74"}
Nov 02 14:17:18.826703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.826141    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:18.828006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.827982    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" event={"ID":"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b","Type":"ContainerStarted","Data":"ce3eedc4c4b2537f73d5d12fd1fc6a5f2e44ad57546ffb2aab1f724b87db452a"}
Nov 02 14:17:18.828447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.828262    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" event={"ID":"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b","Type":"ContainerStarted","Data":"9d63ddf6a6d2aee01527f954ff19def49b6e464b5043b18d44584ff2537d9dee"}
Nov 02 14:17:18.828447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.828342    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:18.829384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.829326    4492 generic.go:356] "Generic (PLEG): container finished" podID="04e8f414-bec1-4a11-acef-eac8d19d8b35" containerID="58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89" exitCode=0
Nov 02 14:17:18.829384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.829378    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k" event={"ID":"04e8f414-bec1-4a11-acef-eac8d19d8b35","Type":"ContainerDied","Data":"58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89"}
Nov 02 14:17:18.829491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.829398    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"
Nov 02 14:17:18.829491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.829415    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k" event={"ID":"04e8f414-bec1-4a11-acef-eac8d19d8b35","Type":"ContainerDied","Data":"a24fc5c6d2c6454b91d4347bd33ff9493d051cedadeaac06420a0e0a31f9d1b3"}
Nov 02 14:17:18.829604 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.829494    4492 scope.go:122] "RemoveContainer" containerID="58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89"
Nov 02 14:17:18.902698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.902374    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" podStartSLOduration=1.902336619 podStartE2EDuration="1.902336619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:18.899831348 +0000 UTC m=+1203.443546947" watchObservedRunningTime="2026-11-02 14:17:18.902336619 +0000 UTC m=+1203.446052214"
Nov 02 14:17:18.927310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.927285    4492 scope.go:122] "RemoveContainer" containerID="58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89"
Nov 02 14:17:18.927851 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:17:18.927812    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89\": container with ID starting with 58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89 not found: ID does not exist" containerID="58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89"
Nov 02 14:17:18.927970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.927910    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89"} err="failed to get container status \"58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89\": rpc error: code = NotFound desc = could not find container \"58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89\": container with ID starting with 58814ae34a96d9e8457fcea50189c6805d05071e10c0a7bb8092300a3dfffe89 not found: ID does not exist"
Nov 02 14:17:18.940391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.937531    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" podStartSLOduration=1.9375139890000002 podStartE2EDuration="1.937513989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:18.936802632 +0000 UTC m=+1203.480518227" watchObservedRunningTime="2026-11-02 14:17:18.937513989 +0000 UTC m=+1203.481229585"
Nov 02 14:17:18.997447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:18.997413    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"]
Nov 02 14:17:19.006417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:19.004968    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-6c6f2lf9k"]
Nov 02 14:17:19.586752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:19.586716    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e8f414-bec1-4a11-acef-eac8d19d8b35" path="/var/lib/kubelet/pods/04e8f414-bec1-4a11-acef-eac8d19d8b35/volumes"
Nov 02 14:17:19.700777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:19.700740    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:17:19.700940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:19.700818    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:17:20.627335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:20.627305    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:20.732515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:20.732483    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:20.747106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:20.747078    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:20.754188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:20.752716    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:17:21.872125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:21.872084    4492 generic.go:356] "Generic (PLEG): container finished" podID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerID="961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237" exitCode=0
Nov 02 14:17:21.872717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:21.872165    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" event={"ID":"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3","Type":"ContainerDied","Data":"961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"}
Nov 02 14:17:21.872785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:21.872744    4492 scope.go:122] "RemoveContainer" containerID="961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"
Nov 02 14:17:21.874468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:21.874437    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:21.874571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:21.874550    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:21.874613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:21.874597    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:22.446919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:22.446879    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:22.882897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:22.882801    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" event={"ID":"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3","Type":"ContainerStarted","Data":"44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755"}
Nov 02 14:17:23.606151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:23.606111    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:17:23.607318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:23.607290    4492 patch_prober.go:29] interesting pod/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:17:23.607498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:23.607341    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:17:23.964438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:23.962818    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:24.246428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:24.246319    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:24.606185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:24.606099    4492 patch_prober.go:29] interesting pod/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:17:24.606185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:24.606158    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:17:25.445547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.445482    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:25.605669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.605633    4492 patch_prober.go:29] interesting pod/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:17:25.605849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.605698    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:17:25.883836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.883764    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"]
Nov 02 14:17:25.887594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.887562    4492 cpu_manager.go:414] "containerMap: removing container" podUID="04e8f414-bec1-4a11-acef-eac8d19d8b35" containerName="payload-processing"
Nov 02 14:17:25.887594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.887593    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e8f414-bec1-4a11-acef-eac8d19d8b35" containerName="payload-processing"
Nov 02 14:17:25.887813 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.887801    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e8f414-bec1-4a11-acef-eac8d19d8b35" containerName="payload-processing"
Nov 02 14:17:25.893821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.893796    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:25.925434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.925401    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"]
Nov 02 14:17:25.928486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.928237    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-12b8da-openshift-default-dockercfg-qr7jf\""
Nov 02 14:17:25.970783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:25.970700    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023612    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023665    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023706    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023777    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023839    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023951    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.023989    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.024047    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.024409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.024081    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwwg\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.169200    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178477    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178631    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178681    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178787    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178828    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwwg\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178914    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178938    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.178984    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.209515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.179071    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.343619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.308771    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.343619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.309021    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.343619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.315522    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.343619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.318575    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.345766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.341726    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.354397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.353656    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.376384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.370285    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.376384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.371255    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwwg\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.376384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.371543    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:26.376384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.373877    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.577153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.576935    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:26.607273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.607239    4492 patch_prober.go:29] interesting pod/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:17:26.607930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.607891    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:17:26.674202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.674171    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:17:26.770379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.769951    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:17:26.781000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.779823    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"]
Nov 02 14:17:26.977798 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:26.977769    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b90407_cb9e_400b_a369_e04b59cde7cc.slice/crio-945e507beca65be92dc2a9032fdb20d8b9b602df5130a5210a2db4c5c9f977ef WatchSource:0}: Error finding container 945e507beca65be92dc2a9032fdb20d8b9b602df5130a5210a2db4c5c9f977ef: Status 404 returned error can't find the container with id 945e507beca65be92dc2a9032fdb20d8b9b602df5130a5210a2db4c5c9f977ef
Nov 02 14:17:26.978114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.978089    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"]
Nov 02 14:17:26.979985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.979945    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:26.980088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.980026    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:26.980088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:26.980068    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:27.606225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:27.606189    4492 patch_prober.go:29] interesting pod/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:17:27.606644 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:27.606237    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:17:27.947379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:27.940948    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:27.971868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:27.971748    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" event={"ID":"c7b90407-cb9e-400b-a369-e04b59cde7cc","Type":"ContainerStarted","Data":"99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"}
Nov 02 14:17:27.971868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:27.971818    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" event={"ID":"c7b90407-cb9e-400b-a369-e04b59cde7cc","Type":"ContainerStarted","Data":"945e507beca65be92dc2a9032fdb20d8b9b602df5130a5210a2db4c5c9f977ef"}
Nov 02 14:17:28.072616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.072542    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podStartSLOduration=3.072518647 podStartE2EDuration="3.072518647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:28.026160443 +0000 UTC m=+1212.569876031" watchObservedRunningTime="2026-11-02 14:17:28.072518647 +0000 UTC m=+1212.616234242"
Nov 02 14:17:28.579623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.579579    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:28.586030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.585997    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:28.610147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.610120    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:17:28.611270 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.611251    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:17:28.981917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.981875    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:28.982977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:28.982954    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:17:29.851950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:29.851925    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:17:29.855598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:29.855251    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:17:29.893703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:29.893673    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:30.682889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:30.682852    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:30.863382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:30.859844    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:30.997446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:30.991647    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:31.780624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:31.780558    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2" podUID="e46f9bbc-b432-469e-b915-324b336d4a9d" containerName="payload-processing" containerID="cri-o://4251aa72dec296ab1f4051f855f27fe7ca55bd9857e3aaee1c98cd70ec7540c2" gracePeriod=25
Nov 02 14:17:32.046651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.046558    4492 generic.go:356] "Generic (PLEG): container finished" podID="e46f9bbc-b432-469e-b915-324b336d4a9d" containerID="4251aa72dec296ab1f4051f855f27fe7ca55bd9857e3aaee1c98cd70ec7540c2" exitCode=0
Nov 02 14:17:32.047283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.047255    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2" event={"ID":"e46f9bbc-b432-469e-b915-324b336d4a9d","Type":"ContainerDied","Data":"4251aa72dec296ab1f4051f855f27fe7ca55bd9857e3aaee1c98cd70ec7540c2"}
Nov 02 14:17:32.162037 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.158130    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.209944    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials\") pod \"e46f9bbc-b432-469e-b915-324b336d4a9d\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") "
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.210046    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume\") pod \"e46f9bbc-b432-469e-b915-324b336d4a9d\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") "
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.210175    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume\") pod \"e46f9bbc-b432-469e-b915-324b336d4a9d\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") "
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.210240    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp\") pod \"e46f9bbc-b432-469e-b915-324b336d4a9d\" (UID: \"e46f9bbc-b432-469e-b915-324b336d4a9d\") "
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.210617    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume" pod "e46f9bbc-b432-469e-b915-324b336d4a9d" (UID: "e46f9bbc-b432-469e-b915-324b336d4a9d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.210856    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume" pod "e46f9bbc-b432-469e-b915-324b336d4a9d" (UID: "e46f9bbc-b432-469e-b915-324b336d4a9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.211233    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.211249    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e46f9bbc-b432-469e-b915-324b336d4a9d-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.213457    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp" pod "e46f9bbc-b432-469e-b915-324b336d4a9d" (UID: "e46f9bbc-b432-469e-b915-324b336d4a9d"). InnerVolumeSpecName "kube-api-access-r4skp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:32.218398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.214494    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials" pod "e46f9bbc-b432-469e-b915-324b336d4a9d" (UID: "e46f9bbc-b432-469e-b915-324b336d4a9d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:32.313161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.313062    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4skp\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-kube-api-access-r4skp\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:32.313161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.313106    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e46f9bbc-b432-469e-b915-324b336d4a9d-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:32.680128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.680089    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:32.737801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.737753    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"]
Nov 02 14:17:32.738901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.738873    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e46f9bbc-b432-469e-b915-324b336d4a9d" containerName="payload-processing"
Nov 02 14:17:32.738901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.738899    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46f9bbc-b432-469e-b915-324b336d4a9d" containerName="payload-processing"
Nov 02 14:17:32.739141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.739128    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46f9bbc-b432-469e-b915-324b336d4a9d" containerName="payload-processing"
Nov 02 14:17:32.744695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.744673    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.760426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.760390    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"]
Nov 02 14:17:32.762964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.762792    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f1e1c2-serving-cert\""
Nov 02 14:17:32.804396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.804349    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:32.821222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.821185    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.821564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.821540    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mbn4\" (UniqueName: \"kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.821687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.821652    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.821770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.821731    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.838460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.838427    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"]
Nov 02 14:17:32.877844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.877804    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"]
Nov 02 14:17:32.878035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.877949    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:32.891098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.891073    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-12b8da\""
Nov 02 14:17:32.919997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.919964    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:32.923295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.923265    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"]
Nov 02 14:17:32.923451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.923421    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.923508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.923484    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.923741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.923723    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mbn4\" (UniqueName: \"kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.923831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.923794    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.924322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.924297    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.926703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.926684    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.926818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.926797    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:32.928316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.928299    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:32.943141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.943083    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-12b8da-dockercfg-dgvh2\""
Nov 02 14:17:32.947737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.947711    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"]
Nov 02 14:17:32.951957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:32.951932    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mbn4\" (UniqueName: \"kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4\") pod \"maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:33.025184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.025140    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.025393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.025214    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5sl\" (UniqueName: \"kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl\") pod \"payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.025393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.025313    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnpm\" (UniqueName: \"kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm\") pod \"payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.025496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.025471    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.034787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.034761    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:33.055512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.055484    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"
Nov 02 14:17:33.055929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.055510    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2" event={"ID":"e46f9bbc-b432-469e-b915-324b336d4a9d","Type":"ContainerDied","Data":"baec51994bbbcf4438f3ae0147a0403a144fba76a07326635ac9381a3d51ee30"}
Nov 02 14:17:33.055929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.055585    4492 scope.go:122] "RemoveContainer" containerID="4251aa72dec296ab1f4051f855f27fe7ca55bd9857e3aaee1c98cd70ec7540c2"
Nov 02 14:17:33.074008 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.073984    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:33.132670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.128035    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.132670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.128275    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.132670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.128333    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5sl\" (UniqueName: \"kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl\") pod \"payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.132670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.128422    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnpm\" (UniqueName: \"kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm\") pod \"payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.150380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.145693    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.150380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.147292    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.204484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.196311    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5sl\" (UniqueName: \"kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl\") pod \"payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.225507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.225467    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnpm\" (UniqueName: \"kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm\") pod \"payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.225803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.225768    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"]
Nov 02 14:17:33.258941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.258904    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-9b79jdvq2"]
Nov 02 14:17:33.293701 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.293664    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:33.496642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.496543    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:33.585466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.585437    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46f9bbc-b432-469e-b915-324b336d4a9d" path="/var/lib/kubelet/pods/e46f9bbc-b432-469e-b915-324b336d4a9d/volumes"
Nov 02 14:17:33.716059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.715713    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"]
Nov 02 14:17:33.723254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.723217    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"]
Nov 02 14:17:33.815333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:33.815233    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"]
Nov 02 14:17:33.818064 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:33.818030    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47119e4f_f07a_4c80_a49e_dd9bc05c9659.slice/crio-2b313b038abb6edd0595cea723a14a7934808c23dd36170430dd26ae5c8ab11e WatchSource:0}: Error finding container 2b313b038abb6edd0595cea723a14a7934808c23dd36170430dd26ae5c8ab11e: Status 404 returned error can't find the container with id 2b313b038abb6edd0595cea723a14a7934808c23dd36170430dd26ae5c8ab11e
Nov 02 14:17:34.085437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.085050    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" event={"ID":"00fd286f-4693-4acc-9087-588159f7198d","Type":"ContainerStarted","Data":"677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26"}
Nov 02 14:17:34.085437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.085120    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" event={"ID":"00fd286f-4693-4acc-9087-588159f7198d","Type":"ContainerStarted","Data":"f43a7d2e59b7ca7de293b46bb9040877bd6566949e9cd0b664f8d7c76902858c"}
Nov 02 14:17:34.085437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.085417    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:34.107194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.104521    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2" event={"ID":"329d2121-eaad-4b40-af4c-31ec54cd398b","Type":"ContainerStarted","Data":"6bc168fa48429cc1b49da3d6690fa45a6c494787c9dbac132194d95b942e3e0b"}
Nov 02 14:17:34.109192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.107762    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" event={"ID":"47119e4f-f07a-4c80-a49e-dd9bc05c9659","Type":"ContainerStarted","Data":"34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164"}
Nov 02 14:17:34.109192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.107820    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" event={"ID":"47119e4f-f07a-4c80-a49e-dd9bc05c9659","Type":"ContainerStarted","Data":"2b313b038abb6edd0595cea723a14a7934808c23dd36170430dd26ae5c8ab11e"}
Nov 02 14:17:34.109192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.108661    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:34.136006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.135943    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" podStartSLOduration=2.135906979 podStartE2EDuration="2.135906979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:34.129298717 +0000 UTC m=+1218.673014329" watchObservedRunningTime="2026-11-02 14:17:34.135906979 +0000 UTC m=+1218.679622573"
Nov 02 14:17:34.171312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.171268    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" podStartSLOduration=2.171252865 podStartE2EDuration="2.171252865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:34.167725825 +0000 UTC m=+1218.711441420" watchObservedRunningTime="2026-11-02 14:17:34.171252865 +0000 UTC m=+1218.714968490"
Nov 02 14:17:34.841344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:34.841310    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:35.118689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:35.118575    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2" event={"ID":"329d2121-eaad-4b40-af4c-31ec54cd398b","Type":"ContainerStarted","Data":"e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc"}
Nov 02 14:17:35.118689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:35.118638    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:35.178761 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:35.178694    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2" podStartSLOduration=2.733686131 podStartE2EDuration="3.178670534s" totalImagesPullingTime="444.984403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:35.161528973 +0000 UTC m=+1219.705244574" watchObservedRunningTime="2026-11-02 14:17:35.178670534 +0000 UTC m=+1219.722386129"
Nov 02 14:17:35.628210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:35.628175    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:36.625623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:36.625587    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:38.534503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:38.534440    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:40.325415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:40.325386    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:41.006625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:41.006593    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"]
Nov 02 14:17:41.048827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:41.048787    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"]
Nov 02 14:17:41.135934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:41.135905    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:17:41.159759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:41.159723    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:41.273244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:41.273166    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:42.231705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:42.231668    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:42.350078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:42.349461    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:44.076439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:44.076379    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:45.122402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:45.122343    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:17:46.007920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.007880    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph" podUID="fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" containerName="payload-processing" containerID="cri-o://030991b4fb99870926fce2ae59e80dced6f6216e9d6d25d8a33bbbc093386b31" gracePeriod=25
Nov 02 14:17:46.049511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.049475    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk" podUID="dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" containerName="payload-pre-processing" containerID="cri-o://c9701f0cc852485ce73ed37ee1fe50badbe278f3ea55db9e5ce0778b0c666c11" gracePeriod=25
Nov 02 14:17:46.131031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.131001    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:17:46.202205 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.202176    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:46.351626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.351531    4492 generic.go:356] "Generic (PLEG): container finished" podID="fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" containerID="030991b4fb99870926fce2ae59e80dced6f6216e9d6d25d8a33bbbc093386b31" exitCode=0
Nov 02 14:17:46.351968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.351944    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph" event={"ID":"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f","Type":"ContainerDied","Data":"030991b4fb99870926fce2ae59e80dced6f6216e9d6d25d8a33bbbc093386b31"}
Nov 02 14:17:46.372970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.372859    4492 generic.go:356] "Generic (PLEG): container finished" podID="dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" containerID="c9701f0cc852485ce73ed37ee1fe50badbe278f3ea55db9e5ce0778b0c666c11" exitCode=0
Nov 02 14:17:46.373301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.373233    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk" event={"ID":"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c","Type":"ContainerDied","Data":"c9701f0cc852485ce73ed37ee1fe50badbe278f3ea55db9e5ce0778b0c666c11"}
Nov 02 14:17:46.434535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.434506    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:17:46.504477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.504436    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58\" (UniqueName: \"kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58\") pod \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") "
Nov 02 14:17:46.504727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.504502    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume\") pod \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\" (UID: \"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c\") "
Nov 02 14:17:46.505175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.505152    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume" pod "dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" (UID: "dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:46.507319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.507288    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58" pod "dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" (UID: "dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c"). InnerVolumeSpecName "kube-api-access-4xj58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:46.517272 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.517247    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:17:46.525667 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:17:46.525628    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:17:46.605617 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.605525    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume\") pod \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") "
Nov 02 14:17:46.605788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.605761    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79\" (UniqueName: \"kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79\") pod \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\" (UID: \"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f\") "
Nov 02 14:17:46.607026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.606998    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xj58\" (UniqueName: \"kubernetes.io/projected/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-kube-api-access-4xj58\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:46.607026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.607028    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:46.607217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.607158    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume" pod "fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" (UID: "fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:46.610234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.609504    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79" pod "fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" (UID: "fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f"). InnerVolumeSpecName "kube-api-access-8fn79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:46.708872 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.708840    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:46.708872 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:46.708870    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fn79\" (UniqueName: \"kubernetes.io/projected/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f-kube-api-access-8fn79\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:47.384623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.384524    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph" event={"ID":"fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f","Type":"ContainerDied","Data":"c5061ed27b81ade04951c9176c72546dd54df68eff0c4be03c36031be00f98f3"}
Nov 02 14:17:47.385083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.384664    4492 scope.go:122] "RemoveContainer" containerID="030991b4fb99870926fce2ae59e80dced6f6216e9d6d25d8a33bbbc093386b31"
Nov 02 14:17:47.385083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.384876    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"
Nov 02 14:17:47.387413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.387333    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk" event={"ID":"dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c","Type":"ContainerDied","Data":"4efacf1b965ce9ed8a518ade22ddead6aaa34013a0eb3e127bd1e5e2f802036b"}
Nov 02 14:17:47.387736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.387720    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"
Nov 02 14:17:47.399579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.399561    4492 scope.go:122] "RemoveContainer" containerID="c9701f0cc852485ce73ed37ee1fe50badbe278f3ea55db9e5ce0778b0c666c11"
Nov 02 14:17:47.436223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.436191    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"]
Nov 02 14:17:47.445954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.445926    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-514524-5cd594dcc9-hctph"]
Nov 02 14:17:47.488158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.488128    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"]
Nov 02 14:17:47.492647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.492622    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-514524-75468d84dc-ffggk"]
Nov 02 14:17:47.584767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.584739    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" path="/var/lib/kubelet/pods/dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c/volumes"
Nov 02 14:17:47.585237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.585219    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" path="/var/lib/kubelet/pods/fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f/volumes"
Nov 02 14:17:47.715565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.715047    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"]
Nov 02 14:17:47.715565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.715290    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf" podUID="098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" containerName="maas-api" containerID="cri-o://72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3" gracePeriod=30
Nov 02 14:17:47.781991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:47.781957    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:48.276379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.275443    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:17:48.360437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.359872    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc\" (UniqueName: \"kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc\") pod \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") "
Nov 02 14:17:48.360437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.359936    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca\") pod \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") "
Nov 02 14:17:48.360437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.359964    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls\") pod \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") "
Nov 02 14:17:48.360437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.360256    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs\") pod \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\" (UID: \"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba\") "
Nov 02 14:17:48.393461 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.362435    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca" pod "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" (UID: "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:48.416242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.414873    4492 generic.go:356] "Generic (PLEG): container finished" podID="098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" containerID="72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3" exitCode=0
Nov 02 14:17:48.416242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.414994    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf" event={"ID":"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba","Type":"ContainerDied","Data":"72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3"}
Nov 02 14:17:48.416242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.415044    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf" event={"ID":"098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba","Type":"ContainerDied","Data":"3a76be60d34300831d866f027fa49e545497e8dc913ad1151d09bf1e6eca5272"}
Nov 02 14:17:48.416242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.415077    4492 scope.go:122] "RemoveContainer" containerID="72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3"
Nov 02 14:17:48.416242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.415189    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"
Nov 02 14:17:48.418830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.418749    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs" pod "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" (UID: "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:17:48.419389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.419095    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc" pod "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" (UID: "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba"). InnerVolumeSpecName "kube-api-access-cgcmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:48.419389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.419171    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls" pod "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" (UID: "098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:17:48.431273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.431248    4492 scope.go:122] "RemoveContainer" containerID="72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3"
Nov 02 14:17:48.431640 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:17:48.431617    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3\": container with ID starting with 72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3 not found: ID does not exist" containerID="72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3"
Nov 02 14:17:48.431733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.431646    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3"} err="failed to get container status \"72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3\": rpc error: code = NotFound desc = could not find container \"72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3\": container with ID starting with 72776c1743306ad5f657ad1b805dd87afa2bed0db1da194508be829be1ac8ed3 not found: ID does not exist"
Nov 02 14:17:48.476493 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.476458    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:48.476493 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.476497    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcmc\" (UniqueName: \"kubernetes.io/projected/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-kube-api-access-cgcmc\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:48.476785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.476513    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:48.476785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.476530    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:48.492433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.492398    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"]
Nov 02 14:17:48.493144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493123    4492 cpu_manager.go:414] "containerMap: removing container" podUID="098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" containerName="maas-api"
Nov 02 14:17:48.493255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493146    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" containerName="maas-api"
Nov 02 14:17:48.493255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493197    4492 cpu_manager.go:414] "containerMap: removing container" podUID="fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" containerName="payload-processing"
Nov 02 14:17:48.493255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493205    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" containerName="payload-processing"
Nov 02 14:17:48.493255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493244    4492 cpu_manager.go:414] "containerMap: removing container" podUID="dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" containerName="payload-pre-processing"
Nov 02 14:17:48.493255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493252    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" containerName="payload-pre-processing"
Nov 02 14:17:48.493539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493422    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7c71bb-bed4-4eb1-98df-055d2bf3fb4f" containerName="payload-processing"
Nov 02 14:17:48.493539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493443    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6c6604-cea8-4fd1-b4e8-f2bd40ac432c" containerName="payload-pre-processing"
Nov 02 14:17:48.493539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.493454    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" containerName="maas-api"
Nov 02 14:17:48.498509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.498487    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.516108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.516081    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"]
Nov 02 14:17:48.522660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.521588    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-12b8da-serving-cert\""
Nov 02 14:17:48.578324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.578288    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.578735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.578712    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.578967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.578950    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6gwh\" (UniqueName: \"kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.579175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.579160    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.584559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.584525    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:48.689371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.687958    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.689371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.688102    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6gwh\" (UniqueName: \"kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.689371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.688307    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.689371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.688484    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.732137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.730725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:48.732137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.731791    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.741383 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:17:48.737561    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-serving-cert: secret "maas-api-e2e-extmod-b-12b8da-serving-cert" not found
Nov 02 14:17:48.741383 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:17:48.737634    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls podName:a2ea0d2a-a00d-4644-bf30-ba64ef76d052 nodeName:}" failed. No retries permitted until 2026-11-02 14:17:49.23761266 +0000 UTC m=+1233.781328248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls") pod "maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" (UID: "a2ea0d2a-a00d-4644-bf30-ba64ef76d052") : secret "maas-api-e2e-extmod-b-12b8da-serving-cert" not found
Nov 02 14:17:48.763387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.762076    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6gwh\" (UniqueName: \"kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.778504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.778112    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:48.831943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.831907    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"]
Nov 02 14:17:48.842639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.842606    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-514524-78c849544c-7ntjf"]
Nov 02 14:17:48.896522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:48.896490    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:49.038683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.038597    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:49.161967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.161929    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:49.267718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.267688    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:49.270510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.270482    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:49.413344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.413266    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:49.587379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.587323    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba" path="/var/lib/kubelet/pods/098f1ab5-ff77-47f3-b3dd-34d5aaddf9ba/volumes"
Nov 02 14:17:49.753095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.753066    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"]
Nov 02 14:17:49.755413 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:49.755384    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ea0d2a_a00d_4644_bf30_ba64ef76d052.slice/crio-2427dcaca10d1024580b487d4dacb25c6b7d7547d85f890696d4e4924b33aa80 WatchSource:0}: Error finding container 2427dcaca10d1024580b487d4dacb25c6b7d7547d85f890696d4e4924b33aa80: Status 404 returned error can't find the container with id 2427dcaca10d1024580b487d4dacb25c6b7d7547d85f890696d4e4924b33aa80
Nov 02 14:17:49.972071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:49.972038    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:50.346338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.346312    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"]
Nov 02 14:17:50.346584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.346546    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy" containerID="cri-o://db613db5ded6c87ae233b5da985d44c9ff6b9bf90c235e4655cd033753fc4403" gracePeriod=30
Nov 02 14:17:50.371125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.371089    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:50.587468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.578150    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" event={"ID":"a2ea0d2a-a00d-4644-bf30-ba64ef76d052","Type":"ContainerStarted","Data":"e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1"}
Nov 02 14:17:50.587468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.578227    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" event={"ID":"a2ea0d2a-a00d-4644-bf30-ba64ef76d052","Type":"ContainerStarted","Data":"2427dcaca10d1024580b487d4dacb25c6b7d7547d85f890696d4e4924b33aa80"}
Nov 02 14:17:50.587468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.579300    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:50.628536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.628455    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" podStartSLOduration=2.146489413 podStartE2EDuration="2.628434638s" totalImagesPullingTime="481.945225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:50.628204998 +0000 UTC m=+1235.171920614" watchObservedRunningTime="2026-11-02 14:17:50.628434638 +0000 UTC m=+1235.172150233"
Nov 02 14:17:50.879844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:50.879732    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:51.376859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:51.376826    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:51.496586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:51.496554    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:51.610598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:51.610548    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:52.742871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:52.742841    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:54.688852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:54.688818    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:55.623147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.623065    4492 generic.go:356] "Generic (PLEG): container finished" podID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerID="db613db5ded6c87ae233b5da985d44c9ff6b9bf90c235e4655cd033753fc4403" exitCode=0
Nov 02 14:17:55.623304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.623151    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" event={"ID":"00dcb0fb-68a0-4605-b953-5426cb18014a","Type":"ContainerDied","Data":"db613db5ded6c87ae233b5da985d44c9ff6b9bf90c235e4655cd033753fc4403"}
Nov 02 14:17:55.623304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.623235    4492 scope.go:122] "RemoveContainer" containerID="87581e28fba08cd11cdff6a2c24e1d26b44448e651bc4709c998c79e1a0b2e7f"
Nov 02 14:17:55.673105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.673083    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:17:55.770278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.770250    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:55.776635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776598    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776656    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776694    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776721    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776801    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776852    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776885    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.776978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776925    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:55.776978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.776937    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.777295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.777007    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data\") pod \"00dcb0fb-68a0-4605-b953-5426cb18014a\" (UID: \"00dcb0fb-68a0-4605-b953-5426cb18014a\") "
Nov 02 14:17:55.777671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.777645    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778049    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778142    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778199    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778490    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778511    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778526    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778538    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.778579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.778550    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00dcb0fb-68a0-4605-b953-5426cb18014a-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.786824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.786684    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:17:55.786824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.786692    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:17:55.786824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.786801    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:55.788511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.788484    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w" pod "00dcb0fb-68a0-4605-b953-5426cb18014a" (UID: "00dcb0fb-68a0-4605-b953-5426cb18014a"). InnerVolumeSpecName "kube-api-access-t7k6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:55.879843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.879758    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.879843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.879786    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.879843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.879799    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00dcb0fb-68a0-4605-b953-5426cb18014a-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:55.879843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:55.879812    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7k6w\" (UniqueName: \"kubernetes.io/projected/00dcb0fb-68a0-4605-b953-5426cb18014a-kube-api-access-t7k6w\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:17:56.275487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.275449    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:56.514165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.514133    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"]
Nov 02 14:17:56.514985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.514964    4492 cpu_manager.go:414] "containerMap: removing container" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy"
Nov 02 14:17:56.514985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.514984    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy"
Nov 02 14:17:56.515143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.515107    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy"
Nov 02 14:17:56.515143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.515121    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy"
Nov 02 14:17:56.519477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.519456    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.546899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.546868    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"]
Nov 02 14:17:56.589304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589268    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589321    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589460    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589496    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589554    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589640    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqws2\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589678    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589743    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.589855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.589780    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.632823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.632788    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"
Nov 02 14:17:56.633011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.632851    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf" event={"ID":"00dcb0fb-68a0-4605-b953-5426cb18014a","Type":"ContainerDied","Data":"b064d051738dcd7fbd38ec8ddd27f0356119b5133ff5f4ba120d4c91cc760cbc"}
Nov 02 14:17:56.633011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.632917    4492 scope.go:122] "RemoveContainer" containerID="db613db5ded6c87ae233b5da985d44c9ff6b9bf90c235e4655cd033753fc4403"
Nov 02 14:17:56.682724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.682692    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"]
Nov 02 14:17:56.690751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.690717    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.690923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.690810    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.690923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.690866    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.690942    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.690970    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqws2\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691013    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691211 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691090    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691211 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691132    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691291    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691329    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691489    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691648 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691604    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.691923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691896    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.692024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.691960    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.692997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.692976    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-514524-openshift-default-5b86c4994f-jdzkf"]
Nov 02 14:17:56.693501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.693484    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.693585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.693546    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.713253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.713220    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.713457 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.713235    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqws2\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.725440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.723089    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:56.843435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.839619    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:56.871377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.863717    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"]
Nov 02 14:17:56.871377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.865088    4492 cpu_manager.go:414] "containerMap: removing container" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy"
Nov 02 14:17:56.871377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.865112    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" containerName="istio-proxy"
Nov 02 14:17:56.912376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.908337    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:56.923027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.922670    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:56.926095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:56.926067    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"]
Nov 02 14:17:57.010331 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.006561    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-623b4b-openshift-default-dockercfg-bw4ld\""
Nov 02 14:17:57.020136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.020112    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.020379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.020344    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.020601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.020583    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.020929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.020907    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.021681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.021658    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.021968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.021950    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.022215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.022200    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.022386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.022343    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.022999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.022968    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwks9\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125172    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125221    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125472    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125510    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125541    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwks9\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125581    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125618    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125647    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.125854    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.126906    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.127166    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.127900    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.128425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.128226    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.130470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.130424    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.141264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.141197    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.143063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.143010    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.159212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.159123    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.159212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.159133    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwks9\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.326334 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.326048    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:57.383003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.382916    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:57.475680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.475641    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"]
Nov 02 14:17:57.481545 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:57.481168    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a57a84_8ac8_4be1_acca_75a3319737c2.slice/crio-83e8c8bf24358be1f3fa18154cfc8994268905e6e589b57cdc61fcff71664c0c WatchSource:0}: Error finding container 83e8c8bf24358be1f3fa18154cfc8994268905e6e589b57cdc61fcff71664c0c: Status 404 returned error can't find the container with id 83e8c8bf24358be1f3fa18154cfc8994268905e6e589b57cdc61fcff71664c0c
Nov 02 14:17:57.484131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.484096    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:57.484247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.484178    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:57.484247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.484224    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:57.584931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.584894    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00dcb0fb-68a0-4605-b953-5426cb18014a" path="/var/lib/kubelet/pods/00dcb0fb-68a0-4605-b953-5426cb18014a/volumes"
Nov 02 14:17:57.601549 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.601522    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:17:57.646997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.646893    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" event={"ID":"77a57a84-8ac8-4be1-acca-75a3319737c2","Type":"ContainerStarted","Data":"175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1"}
Nov 02 14:17:57.646997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.646963    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" event={"ID":"77a57a84-8ac8-4be1-acca-75a3319737c2","Type":"ContainerStarted","Data":"83e8c8bf24358be1f3fa18154cfc8994268905e6e589b57cdc61fcff71664c0c"}
Nov 02 14:17:57.731448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.731215    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"]
Nov 02 14:17:57.735702 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:17:57.735677    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677cda29_e1cc_4b89_b851_9460141412a7.slice/crio-879469389d50ce09c290a1f1a77fbecf9dbb39eba984bf205825898e0c286f18 WatchSource:0}: Error finding container 879469389d50ce09c290a1f1a77fbecf9dbb39eba984bf205825898e0c286f18: Status 404 returned error can't find the container with id 879469389d50ce09c290a1f1a77fbecf9dbb39eba984bf205825898e0c286f18
Nov 02 14:17:57.739552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.739508    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:57.739805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.739777    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:57.739919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.739841    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:17:57.841549 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:57.841518    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:17:58.673023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:58.672984    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" event={"ID":"677cda29-e1cc-4b89-b851-9460141412a7","Type":"ContainerStarted","Data":"d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"}
Nov 02 14:17:58.674166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:58.673045    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" event={"ID":"677cda29-e1cc-4b89-b851-9460141412a7","Type":"ContainerStarted","Data":"879469389d50ce09c290a1f1a77fbecf9dbb39eba984bf205825898e0c286f18"}
Nov 02 14:17:58.722281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:58.722224    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podStartSLOduration=2.722210111 podStartE2EDuration="2.722210111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:58.721147796 +0000 UTC m=+1243.264863389" watchObservedRunningTime="2026-11-02 14:17:58.722210111 +0000 UTC m=+1243.265925705"
Nov 02 14:17:58.724557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:58.724516    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podStartSLOduration=2.72450403 podStartE2EDuration="2.72450403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:57.731562888 +0000 UTC m=+1242.275278482" watchObservedRunningTime="2026-11-02 14:17:58.72450403 +0000 UTC m=+1243.268219624"
Nov 02 14:17:58.876579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:58.876490    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 14:17:58.876833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:58.876804    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 14:17:59.327317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.327279    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:59.342714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.342678    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:59.463393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.456482    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:17:59.683198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.683176    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:59.684576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.684555    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:17:59.865089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.865055    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 14:17:59.865295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:17:59.865113    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 14:18:00.849705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:00.848494    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 14:18:00.849705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:00.848560    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 14:18:01.264607 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:01.264471    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:01.388682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:01.388646    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:01.700796 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:01.700757    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:01.848379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:01.843501    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 14:18:01.848379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:01.843574    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 14:18:02.855382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:02.854278    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 14:18:02.855382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:02.854400    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 14:18:03.040387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:03.032454    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:03.166060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:03.165957    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:03.739532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:03.739504    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:03.843107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:03.843070    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 14:18:03.843481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:03.843444    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 14:18:05.346242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:05.346209    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:18:05.347708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:05.347684    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:18:05.348600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:05.348577    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:18:05.533205 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:05.533173    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:07.224666 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:07.224621    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:07.549113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:07.548992    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:09.557939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:09.557904    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:11.327857 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:11.327809    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:12.086299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:12.086237    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:13.232569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:13.232531    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:14.976270 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:14.976233    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:15.026912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:15.026507    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:18:15.026912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:15.026592    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:18:16.369618 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.369528    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:16.525047 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:18:16.525010    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:18:16.745664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.745631    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:16.882818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.882788    4492 generic.go:356] "Generic (PLEG): container finished" podID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerID="99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05" exitCode=0
Nov 02 14:18:16.882985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.882860    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" event={"ID":"c7b90407-cb9e-400b-a369-e04b59cde7cc","Type":"ContainerDied","Data":"99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"}
Nov 02 14:18:16.883390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.883334    4492 scope.go:122] "RemoveContainer" containerID="99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"
Nov 02 14:18:16.885013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.884980    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:16.885101 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.885073    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:16.885141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:16.885105    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:17.662439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:17.660966    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:17.781056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:17.781013    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:17.894237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:17.894202    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" event={"ID":"c7b90407-cb9e-400b-a369-e04b59cde7cc","Type":"ContainerStarted","Data":"9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e"}
Nov 02 14:18:17.983867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:17.983834    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:18.307019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:18.306940    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:18.579559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:18.579465    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:18:18.579826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:18.579795    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 14:18:18.579973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:18.579857    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 14:18:18.749380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:18.739838    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:18.997319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:18.997071    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:19.126896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:19.126867    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:19.633379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:19.620295    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 14:18:19.633379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:19.620385    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 14:18:20.417607 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.417578    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:20.458378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.457536    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"]
Nov 02 14:18:20.463926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.463899    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.486611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.486586    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"]
Nov 02 14:18:20.490842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.490819    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-623b4b-serving-cert\""
Nov 02 14:18:20.523281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.523242    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.523712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.523693    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtktf\" (UniqueName: \"kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.524058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.524041    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.524285 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.524269    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.567753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.567718    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:20.579274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.579239    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 14:18:20.579520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.579304    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 14:18:20.625277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.625245    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.625481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.625339    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtktf\" (UniqueName: \"kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.625481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.625424    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.625599 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.625519    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.625713 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:18:20.625694    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-serving-cert: secret "maas-api-e2e-shared-b-w4-623b4b-serving-cert" not found
Nov 02 14:18:20.625780 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:18:20.625767    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls podName:de86f7d7-e096-429a-96d7-828c085393c9 nodeName:}" failed. No retries permitted until 2026-11-02 14:18:21.125747806 +0000 UTC m=+1265.669463397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls") pod "maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" (UID: "de86f7d7-e096-429a-96d7-828c085393c9") : secret "maas-api-e2e-shared-b-w4-623b4b-serving-cert" not found
Nov 02 14:18:20.625954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.625925    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.628312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.628288    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.663981 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.663947    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtktf\" (UniqueName: \"kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:20.691531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.691498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:20.813381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:20.805545    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:21.164317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:21.164217    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:21.167158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:21.167122    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:21.398382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:21.396611    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:21.586542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:21.586508    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 14:18:21.587003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:21.586557    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 14:18:21.849274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:21.849190    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"]
Nov 02 14:18:21.873822 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:18:21.873793    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde86f7d7_e096_429a_96d7_828c085393c9.slice/crio-20a9ff2188b009cf8ef4e464e9b1b40da4b1621c38104766615d9a7480d588f9 WatchSource:0}: Error finding container 20a9ff2188b009cf8ef4e464e9b1b40da4b1621c38104766615d9a7480d588f9: Status 404 returned error can't find the container with id 20a9ff2188b009cf8ef4e464e9b1b40da4b1621c38104766615d9a7480d588f9
Nov 02 14:18:22.104395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:22.104232    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" event={"ID":"de86f7d7-e096-429a-96d7-828c085393c9","Type":"ContainerStarted","Data":"20a9ff2188b009cf8ef4e464e9b1b40da4b1621c38104766615d9a7480d588f9"}
Nov 02 14:18:22.199469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:22.196025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:22.358491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:22.352220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:22.469928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:22.469884    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:22.587764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:22.584714    4492 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 14:18:22.587764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:22.584775    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 14:18:23.115001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.114905    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" event={"ID":"de86f7d7-e096-429a-96d7-828c085393c9","Type":"ContainerStarted","Data":"ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359"}
Nov 02 14:18:23.115182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.115037    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:23.166748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.166720    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"]
Nov 02 14:18:23.176273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.176195    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"]
Nov 02 14:18:23.177366 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.177216    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" podStartSLOduration=2.694003159 podStartE2EDuration="3.177198636s" totalImagesPullingTime="483.195477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:23.166577594 +0000 UTC m=+1267.710293188" watchObservedRunningTime="2026-11-02 14:18:23.177198636 +0000 UTC m=+1267.720914231"
Nov 02 14:18:23.188789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.188760    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.196495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.196464    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"]
Nov 02 14:18:23.196767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.196750    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.202863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.202831    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4tfr8\""
Nov 02 14:18:23.203578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.203554    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.208849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.208828    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"]
Nov 02 14:18:23.216059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.216029    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"]
Nov 02 14:18:23.260136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.259170    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"]
Nov 02 14:18:23.280056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.280022    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.280259 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.280188    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.280339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.280269    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfaf084d-3acb-4659-8503-22bcfc06ee89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.280339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.280326    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.280595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.280565    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.281074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.281035    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.281234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.281151    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.281317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.281298    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.281616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.281598    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpb5\" (UniqueName: \"kubernetes.io/projected/cfaf084d-3acb-4659-8503-22bcfc06ee89-kube-api-access-jjpb5\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.394142    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d4191a-d188-4436-b784-b5e376fc5ef1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.398874    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.398928    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399028    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7958p\" (UniqueName: \"kubernetes.io/projected/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-kube-api-access-7958p\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399068    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399099    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399126    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfaf084d-3acb-4659-8503-22bcfc06ee89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399154    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399201    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399241    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399265    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399292    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399333    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399383    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxds\" (UniqueName: \"kubernetes.io/projected/15d4191a-d188-4436-b784-b5e376fc5ef1-kube-api-access-xgxds\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399417    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399419    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.399574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399448    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399726    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399730    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399827    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399873    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399873    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.399953    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400116    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400160    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfaf084d-3acb-4659-8503-22bcfc06ee89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400191    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400221    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpb5\" (UniqueName: \"kubernetes.io/projected/cfaf084d-3acb-4659-8503-22bcfc06ee89-kube-api-access-jjpb5\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400256    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400438    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400488    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.400700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400612    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.401454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.400764    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.403083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.403061    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.403283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.403262    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.422620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.422583    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfaf084d-3acb-4659-8503-22bcfc06ee89-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.423117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.423099    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpb5\" (UniqueName: \"kubernetes.io/projected/cfaf084d-3acb-4659-8503-22bcfc06ee89-kube-api-access-jjpb5\") pod \"maas-default-gateway-openshift-default-55fbd65575-zvq6k\" (UID: \"cfaf084d-3acb-4659-8503-22bcfc06ee89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.502483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502448    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.502667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502497    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502528    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502561    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502586    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxds\" (UniqueName: \"kubernetes.io/projected/15d4191a-d188-4436-b784-b5e376fc5ef1-kube-api-access-xgxds\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.502667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502621    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502652    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502674    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.502880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502717    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.502880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502745    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502785    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.502880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502822    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.502880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502852    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.503190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.502914    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.503190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.503139    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.503527    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.503601    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.503893    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.503985    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.504133    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d4191a-d188-4436-b784-b5e376fc5ef1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.504186    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.504293    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7958p\" (UniqueName: \"kubernetes.io/projected/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-kube-api-access-7958p\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.504481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.504398    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.505460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.505116    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.505460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.505341    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.505776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.505755    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.505929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.505901    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.506057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.505810    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15d4191a-d188-4436-b784-b5e376fc5ef1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.506129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.506097    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.506191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.506131    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.506483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.506462    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.506549 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.506526    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.521474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.521449    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:23.525921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.525870    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.540576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.540549    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15d4191a-d188-4436-b784-b5e376fc5ef1-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.540784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.540760    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxds\" (UniqueName: \"kubernetes.io/projected/15d4191a-d188-4436-b784-b5e376fc5ef1-kube-api-access-xgxds\") pod \"maas-default-gateway-openshift-default-55fbd65575-89nq9\" (UID: \"15d4191a-d188-4436-b784-b5e376fc5ef1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.543750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.543696    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7958p\" (UniqueName: \"kubernetes.io/projected/6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c-kube-api-access-7958p\") pod \"maas-default-gateway-openshift-default-55fbd65575-zdbqj\" (UID: \"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.559758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.559724    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:23.570263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.570233    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:23.874301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.874263    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"]
Nov 02 14:18:23.874842 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:18:23.874812    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaf084d_3acb_4659_8503_22bcfc06ee89.slice/crio-e0f92187a0588bd9dd25100581b868a651023ca4826f706e9383ee1d56fd37df WatchSource:0}: Error finding container e0f92187a0588bd9dd25100581b868a651023ca4826f706e9383ee1d56fd37df: Status 404 returned error can't find the container with id e0f92187a0588bd9dd25100581b868a651023ca4826f706e9383ee1d56fd37df
Nov 02 14:18:23.877795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.877755    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.877873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.877846    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.878895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.878863    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.907523 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.907446    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:18:23.908877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.908855    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:18:23.912433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.912409    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"]
Nov 02 14:18:23.912837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.912787    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:18:23.914476 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:18:23.914432    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d4191a_d188_4436_b784_b5e376fc5ef1.slice/crio-e81814f32427a14c27bce0b0d3fdb617fdd37d361e72280f24e64a04e31e804f WatchSource:0}: Error finding container e81814f32427a14c27bce0b0d3fdb617fdd37d361e72280f24e64a04e31e804f: Status 404 returned error can't find the container with id e81814f32427a14c27bce0b0d3fdb617fdd37d361e72280f24e64a04e31e804f
Nov 02 14:18:23.917420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.917345    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.917532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.917471    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.917588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.917534    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.932704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.932670    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"]
Nov 02 14:18:23.936122 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:18:23.935057    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa38098_b6ca_4a0b_a5f1_7fafa2bda22c.slice/crio-176c0a09b24678c6132fa5168c346987330cec8f59e0eee0ec4e122b61b1ab18 WatchSource:0}: Error finding container 176c0a09b24678c6132fa5168c346987330cec8f59e0eee0ec4e122b61b1ab18: Status 404 returned error can't find the container with id 176c0a09b24678c6132fa5168c346987330cec8f59e0eee0ec4e122b61b1ab18
Nov 02 14:18:23.938972 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.938933    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.939094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.939033    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:23.939094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:23.939074    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:24.168627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.168542    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" event={"ID":"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c","Type":"ContainerStarted","Data":"9fda46fd1769b3fd8057232f7255bc540fd10668c5281ddb521ff825a8786005"}
Nov 02 14:18:24.168862 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.168839    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" event={"ID":"6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c","Type":"ContainerStarted","Data":"176c0a09b24678c6132fa5168c346987330cec8f59e0eee0ec4e122b61b1ab18"}
Nov 02 14:18:24.174294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.174187    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" event={"ID":"cfaf084d-3acb-4659-8503-22bcfc06ee89","Type":"ContainerStarted","Data":"de6c6813d0242578eec4fd31033f68cb75807c0176bb623c9093e814348ec212"}
Nov 02 14:18:24.174294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.174248    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" event={"ID":"cfaf084d-3acb-4659-8503-22bcfc06ee89","Type":"ContainerStarted","Data":"e0f92187a0588bd9dd25100581b868a651023ca4826f706e9383ee1d56fd37df"}
Nov 02 14:18:24.197263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.197205    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" event={"ID":"15d4191a-d188-4436-b784-b5e376fc5ef1","Type":"ContainerStarted","Data":"4f69deb5dde489291f958de90a836f0a14c1089b03e8beeb5f7a09bbe687f93b"}
Nov 02 14:18:24.199247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.199218    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" event={"ID":"15d4191a-d188-4436-b784-b5e376fc5ef1","Type":"ContainerStarted","Data":"e81814f32427a14c27bce0b0d3fdb617fdd37d361e72280f24e64a04e31e804f"}
Nov 02 14:18:24.304701 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.304637    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podStartSLOduration=1.3046150029999999 podStartE2EDuration="1.304615003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:24.3000445 +0000 UTC m=+1268.843760094" watchObservedRunningTime="2026-11-02 14:18:24.304615003 +0000 UTC m=+1268.848330598"
Nov 02 14:18:24.305714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.305306    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podStartSLOduration=1.305291976 podStartE2EDuration="1.305291976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:24.245962311 +0000 UTC m=+1268.789677912" watchObservedRunningTime="2026-11-02 14:18:24.305291976 +0000 UTC m=+1268.849007573"
Nov 02 14:18:24.343208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.343150    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podStartSLOduration=1.343132076 podStartE2EDuration="1.343132076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:24.340280226 +0000 UTC m=+1268.883995820" watchObservedRunningTime="2026-11-02 14:18:24.343132076 +0000 UTC m=+1268.886847670"
Nov 02 14:18:24.373508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.373477    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:24.535173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.533190    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:24.535173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.533342    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:24.535173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.533411    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:24.560651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.559925    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:24.562512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.562481    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:24.562690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.562544    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:24.570754 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:24.570726    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:25.522816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:25.522764    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:25.523412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:25.522837    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:25.583382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:25.582714    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:25.583382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:25.582773    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:25.624573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:25.624534    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:25.624859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:25.624832    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:26.480388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.446011    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:26.545697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.545409    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:26.623317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.623268    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:26.623971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.607877    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:26.624151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.624130    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:26.624268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.614237    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:26.624399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:26.624380    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:27.566665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:27.556974    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:27.566665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:27.557037    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:27.572907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:27.568468    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:27.572907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:27.568525    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:27.581418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:27.573249    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:27.581418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:27.573303    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:28.173379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.173277    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:28.531226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.530718    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:28.531226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.530804    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:28.569380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.569031    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:28.569380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.569094    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:28.581392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.572685    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:28.581392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:28.572737    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:29.225930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.224720    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:18:29.563954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.531022    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:29.563954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.531089    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:29.565476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.564842    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:29.565476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.564912    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:29.640049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.598868    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:29.640049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:29.598922    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:30.147387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.138497    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:30.523311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.522825    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:30.523311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.522892    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:30.583597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.582028    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:30.583597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.582089    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:30.583597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.583067    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:30.583597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:30.583109    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:31.522503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.522461    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:31.523128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.523093    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:31.560292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.560252    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:31.560516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.560328    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:31.570644 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.570609    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:18:31.570800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.570663    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 14:18:31.834866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:31.834781    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:32.522741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:32.522707    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 14:18:32.523276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:32.522766    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 14:18:32.566255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:32.566202    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:18:32.566464 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:32.566264    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 14:18:33.617590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.617526    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:18:33.618837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.618411    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:18:33.642475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.642273    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:33.899588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.899482    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:33.900638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.900614    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:33.902576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.902539    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj"
Nov 02 14:18:33.928596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.926657    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:33.928596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.927335    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:33.930504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.930477    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k"
Nov 02 14:18:33.987975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.987945    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:33.988658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.988632    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:33.989572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:33.989553    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9"
Nov 02 14:18:35.668381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:35.667278    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:37.748381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:37.745043    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:39.658158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:39.657962    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:41.122308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:41.122275    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:41.550591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:41.550546    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:43.360501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:43.360461    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:45.134706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.134669    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:45.555727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.553836    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": read tcp 10.134.0.2:36412->10.134.0.114:15021: read: connection reset by peer" start-of-body=
Nov 02 14:18:45.555727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.553963    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": read tcp 10.134.0.2:36412->10.134.0.114:15021: read: connection reset by peer"
Nov 02 14:18:45.973093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.973059    4492 generic.go:356] "Generic (PLEG): container finished" podID="677cda29-e1cc-4b89-b851-9460141412a7" containerID="d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4" exitCode=0
Nov 02 14:18:45.973282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.973139    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" event={"ID":"677cda29-e1cc-4b89-b851-9460141412a7","Type":"ContainerDied","Data":"d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"}
Nov 02 14:18:45.973627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.973603    4492 scope.go:122] "RemoveContainer" containerID="d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"
Nov 02 14:18:45.974997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.974966    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:45.975090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.975036    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:45.975090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:45.975069    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:18:46.727377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:46.721827    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:46.821395 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:18:46.816001    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:18:46.890224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:46.890108    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:47.030429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:47.025972    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:47.095937 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:47.093095    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" event={"ID":"677cda29-e1cc-4b89-b851-9460141412a7","Type":"ContainerStarted","Data":"ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878"}
Nov 02 14:18:47.229287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:47.229249    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:47.332419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:47.331752    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:18:47.332419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:47.331876    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 14:18:47.332419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:47.331932    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 14:18:48.327528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:48.327488    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 14:18:48.328020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:48.327546    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 14:18:48.970620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:48.970582    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:49.099383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:49.095912    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:49.327710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:49.327627    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 14:18:49.327710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:49.327703    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 14:18:49.812792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:49.812750    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:50.327406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:50.327332    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 14:18:50.327591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:50.327432    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 14:18:50.953382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:50.951993    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:51.327710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:51.327612    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 14:18:51.327710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:51.327696    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 14:18:52.327621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:52.327585    4492 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 02 14:18:52.328139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:52.327646    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 02 14:18:52.838318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:52.838275    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:53.331823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:53.331793    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:18:53.332308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:53.332286    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:18:53.332959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:53.332937    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:18:54.645296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:54.645267    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:55.454019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:55.453981    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:55.570871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:55.569795    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:56.332153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:56.332083    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:58.093586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:58.093549    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:18:59.783433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:59.783390    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"]
Nov 02 14:18:59.939238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:18:59.939198    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:02.026898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:02.026856    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:02.197144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:02.197109    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:04.002347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:04.002313    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:04.784049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:04.784006    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b" podUID="76ba2e01-bfbc-48de-8098-f8cdc2638257" containerName="payload-processing" containerID="cri-o://5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2" gracePeriod=25
Nov 02 14:19:05.093898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.093861    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:19:05.176100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.176064    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n\") pod \"76ba2e01-bfbc-48de-8098-f8cdc2638257\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") "
Nov 02 14:19:05.176692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.176447    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials\") pod \"76ba2e01-bfbc-48de-8098-f8cdc2638257\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") "
Nov 02 14:19:05.176692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.176517    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume\") pod \"76ba2e01-bfbc-48de-8098-f8cdc2638257\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") "
Nov 02 14:19:05.176969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.176943    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume" pod "76ba2e01-bfbc-48de-8098-f8cdc2638257" (UID: "76ba2e01-bfbc-48de-8098-f8cdc2638257"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:05.177423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.177372    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume" pod "76ba2e01-bfbc-48de-8098-f8cdc2638257" (UID: "76ba2e01-bfbc-48de-8098-f8cdc2638257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:05.177423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.176549    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume\") pod \"76ba2e01-bfbc-48de-8098-f8cdc2638257\" (UID: \"76ba2e01-bfbc-48de-8098-f8cdc2638257\") "
Nov 02 14:19:05.178178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.178154    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:05.178274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.178181    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ba2e01-bfbc-48de-8098-f8cdc2638257-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:05.179567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.179540    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials" pod "76ba2e01-bfbc-48de-8098-f8cdc2638257" (UID: "76ba2e01-bfbc-48de-8098-f8cdc2638257"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:05.181341 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.181312    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n" pod "76ba2e01-bfbc-48de-8098-f8cdc2638257" (UID: "76ba2e01-bfbc-48de-8098-f8cdc2638257"). InnerVolumeSpecName "kube-api-access-9x79n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:05.286859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.286819    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x79n\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-kube-api-access-9x79n\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:05.286859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.286859    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ba2e01-bfbc-48de-8098-f8cdc2638257-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:05.356462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.354242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:05.361945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.361883    4492 generic.go:356] "Generic (PLEG): container finished" podID="76ba2e01-bfbc-48de-8098-f8cdc2638257" containerID="5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2" exitCode=0
Nov 02 14:19:05.362104 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.362014    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b" event={"ID":"76ba2e01-bfbc-48de-8098-f8cdc2638257","Type":"ContainerDied","Data":"5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2"}
Nov 02 14:19:05.362186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.362118    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b" event={"ID":"76ba2e01-bfbc-48de-8098-f8cdc2638257","Type":"ContainerDied","Data":"d16e0bfa486001d267779b134dcbf309c9316d4dc855e1c814671ce46637e505"}
Nov 02 14:19:05.362186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.362173    4492 scope.go:122] "RemoveContainer" containerID="5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2"
Nov 02 14:19:05.362704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.362685    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"
Nov 02 14:19:05.393515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.393492    4492 scope.go:122] "RemoveContainer" containerID="5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2"
Nov 02 14:19:05.393881 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:19:05.393854    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2\": container with ID starting with 5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2 not found: ID does not exist" containerID="5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2"
Nov 02 14:19:05.393972 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.393904    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2"} err="failed to get container status \"5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2\": rpc error: code = NotFound desc = could not find container \"5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2\": container with ID starting with 5a576b59bd7222476d60306bb2057b335097c4770bd94cc25edc21b3bd814ba2 not found: ID does not exist"
Nov 02 14:19:05.415007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.414971    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"]
Nov 02 14:19:05.419141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.419109    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-59cc9hn6b"]
Nov 02 14:19:05.465469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.465428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:05.672895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:05.672803    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ba2e01-bfbc-48de-8098-f8cdc2638257" path="/var/lib/kubelet/pods/76ba2e01-bfbc-48de-8098-f8cdc2638257/volumes"
Nov 02 14:19:06.075126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:06.075094    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:06.736425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:06.736383    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:07.393823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:07.393783    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:07.924018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:07.923970    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:09.724955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:09.724903    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:11.534456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:11.534417    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:12.212923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:12.212887    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:13.339720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:13.339680    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:15.138277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:15.138246    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:16.520828 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:19:16.520788    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:19:16.928434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:16.928400    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:18.920017 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:18.919985    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:19.035652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:19.035620    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:20.756421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:20.750833    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:22.778381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:22.772910    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:23.181613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:23.181577    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:25.661693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:25.661663    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:27.539286 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:27.539250    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:29.331401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:29.331334    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:31.123562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:31.123513    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:31.538958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:31.538926    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:31.684420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:31.678299    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:32.867845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:32.867806    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:34.727451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:34.727407    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:36.466955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:36.466917    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:38.322746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:38.322717    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:39.311268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:39.311229    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"]
Nov 02 14:19:39.311524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:39.311489    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy" containerID="cri-o://d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501" gracePeriod=30
Nov 02 14:19:40.014683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:40.014646    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:41.824756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:41.824722    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:42.595051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:42.595018    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:43.709245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:43.709208    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:44.631512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.631482    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:19:44.705116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.705009    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:44.733699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.733663    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.733767    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.733807    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.733857    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.733926    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.733972    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734027    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734122    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734164    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734225    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:44.734773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734269    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:44.734773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734306    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo\") pod \"40d7fbd3-b8d1-42d7-b970-508195d497bd\" (UID: \"40d7fbd3-b8d1-42d7-b970-508195d497bd\") "
Nov 02 14:19:44.734773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734456    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:44.734773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734609    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:44.734982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734951    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.734982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734974    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.735086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.734990    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.735086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.735005    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d7fbd3-b8d1-42d7-b970-508195d497bd-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.735086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.735017    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.737431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.737347    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:19:44.737431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.737405    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:44.737626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.737491    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:44.738833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.738808    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n" pod "40d7fbd3-b8d1-42d7-b970-508195d497bd" (UID: "40d7fbd3-b8d1-42d7-b970-508195d497bd"). InnerVolumeSpecName "kube-api-access-wcx4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:44.857436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.856504    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.857436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.856546    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.857436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.856560    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.857436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.856573    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcx4n\" (UniqueName: \"kubernetes.io/projected/40d7fbd3-b8d1-42d7-b970-508195d497bd-kube-api-access-wcx4n\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:19:44.877381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:44.874962    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:45.023379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.021017    4492 generic.go:356] "Generic (PLEG): container finished" podID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerID="d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501" exitCode=0
Nov 02 14:19:45.023379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.021324    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"
Nov 02 14:19:45.026036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.024590    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" event={"ID":"40d7fbd3-b8d1-42d7-b970-508195d497bd","Type":"ContainerDied","Data":"d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501"}
Nov 02 14:19:45.026036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.024663    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl" event={"ID":"40d7fbd3-b8d1-42d7-b970-508195d497bd","Type":"ContainerDied","Data":"c2c53af22c13bb194dd56969f180d0d39b0d052bc2764f8bff4f0adb7fbd542c"}
Nov 02 14:19:45.026036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.024696    4492 scope.go:122] "RemoveContainer" containerID="d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501"
Nov 02 14:19:45.077000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.076488    4492 scope.go:122] "RemoveContainer" containerID="d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501"
Nov 02 14:19:45.077000 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:19:45.076955    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501\": container with ID starting with d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501 not found: ID does not exist" containerID="d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501"
Nov 02 14:19:45.077227 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.076997    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501"} err="failed to get container status \"d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501\": rpc error: code = NotFound desc = could not find container \"d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501\": container with ID starting with d80b56965514f06e7b0b15479f1007f0daeff529ffaea8722374de5e899c7501 not found: ID does not exist"
Nov 02 14:19:45.105458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.105422    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"]
Nov 02 14:19:45.109631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.109554    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"]
Nov 02 14:19:45.110278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.110260    4492 cpu_manager.go:414] "containerMap: removing container" podUID="76ba2e01-bfbc-48de-8098-f8cdc2638257" containerName="payload-processing"
Nov 02 14:19:45.110278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.110277    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ba2e01-bfbc-48de-8098-f8cdc2638257" containerName="payload-processing"
Nov 02 14:19:45.110462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.110287    4492 cpu_manager.go:414] "containerMap: removing container" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy"
Nov 02 14:19:45.110462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.110292    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy"
Nov 02 14:19:45.110462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.110443    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ba2e01-bfbc-48de-8098-f8cdc2638257" containerName="payload-processing"
Nov 02 14:19:45.110462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.110453    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" containerName="istio-proxy"
Nov 02 14:19:45.115446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.115417    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-7nfbl"]
Nov 02 14:19:45.115591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.115472    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.122318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.122293    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"]
Nov 02 14:19:45.130860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.130836    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"routing-overlay\""
Nov 02 14:19:45.131111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.131094    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"openshift-service-ca.crt\""
Nov 02 14:19:45.131298 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.131285    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"kube-root-ca.crt\""
Nov 02 14:19:45.131563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.131403    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-f650d0393004\""
Nov 02 14:19:45.131563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.131477    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"agc-cred-2a51f04e5b66\""
Nov 02 14:19:45.131563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.131502    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bb4ef\""
Nov 02 14:19:45.131563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.131538    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bb4ef\"/\"payload-processing-external-model-e2e--c0104787-dockercfg-tx5xz\""
Nov 02 14:19:45.269969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.269916    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.270181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.270069    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554bh\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.270222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.270197    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.270277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.270262    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.372402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.372292    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.372807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.372780    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.372999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.372981    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554bh\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.373222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.373199    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.387378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.375239    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.388579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.388192    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.392164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.392131    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.450760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.450723    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554bh\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh\") pod \"payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") " pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.599250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.599208    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d7fbd3-b8d1-42d7-b970-508195d497bd" path="/var/lib/kubelet/pods/40d7fbd3-b8d1-42d7-b970-508195d497bd/volumes"
Nov 02 14:19:45.733049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.733010    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:45.916130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:45.916092    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:46.088680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:46.088640    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"]
Nov 02 14:19:46.088957 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:19:46.088931    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dc61e7_e5c5_48b2_8c88_84e1672a1858.slice/crio-47de42d2a7529d7a2f92aea43c09fff96c45f2b726ec4c496af9b09ffa129a8b WatchSource:0}: Error finding container 47de42d2a7529d7a2f92aea43c09fff96c45f2b726ec4c496af9b09ffa129a8b: Status 404 returned error can't find the container with id 47de42d2a7529d7a2f92aea43c09fff96c45f2b726ec4c496af9b09ffa129a8b
Nov 02 14:19:46.520489 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:19:46.520451    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:19:47.066939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:47.066895    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z" event={"ID":"b3dc61e7-e5c5-48b2-8c88-84e1672a1858","Type":"ContainerStarted","Data":"5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61"}
Nov 02 14:19:47.067496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:47.066991    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z" event={"ID":"b3dc61e7-e5c5-48b2-8c88-84e1672a1858","Type":"ContainerStarted","Data":"47de42d2a7529d7a2f92aea43c09fff96c45f2b726ec4c496af9b09ffa129a8b"}
Nov 02 14:19:47.067496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:47.067126    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:47.109135 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:47.109084    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z" podStartSLOduration=2.109069207 podStartE2EDuration="2.109069207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:47.106319475 +0000 UTC m=+1351.650035070" watchObservedRunningTime="2026-11-02 14:19:47.109069207 +0000 UTC m=+1351.652784800"
Nov 02 14:19:47.828689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:47.828656    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:49.535706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:49.535666    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:51.238733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:51.238702    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:52.926506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:52.926477    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:54.751559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:54.751517    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:56.541929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:56.541896    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:56.688909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:56.688869    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:56.845430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:56.835123    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:19:57.142401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:57.142288    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"]
Nov 02 14:19:57.146098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:57.146071    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:19:58.513224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:19:58.513187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:00.218980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:00.218947    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:01.932692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:01.932655    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:02.143047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.142997    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z" podUID="b3dc61e7-e5c5-48b2-8c88-84e1672a1858" containerName="payload-processing" containerID="cri-o://5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61" gracePeriod=25
Nov 02 14:20:02.425261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.425234    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:20:02.565282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.565179    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume\") pod \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") "
Nov 02 14:20:02.565282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.565262    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume\") pod \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") "
Nov 02 14:20:02.565538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.565525    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials\") pod \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") "
Nov 02 14:20:02.565603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.565555    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh\") pod \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\" (UID: \"b3dc61e7-e5c5-48b2-8c88-84e1672a1858\") "
Nov 02 14:20:02.565659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.565637    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume" pod "b3dc61e7-e5c5-48b2-8c88-84e1672a1858" (UID: "b3dc61e7-e5c5-48b2-8c88-84e1672a1858"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:02.565757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.565731    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume" pod "b3dc61e7-e5c5-48b2-8c88-84e1672a1858" (UID: "b3dc61e7-e5c5-48b2-8c88-84e1672a1858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:02.566146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.566125    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:20:02.566224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.566151    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:20:02.567853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.567826    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials" pod "b3dc61e7-e5c5-48b2-8c88-84e1672a1858" (UID: "b3dc61e7-e5c5-48b2-8c88-84e1672a1858"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:02.567972 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.567860    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh" pod "b3dc61e7-e5c5-48b2-8c88-84e1672a1858" (UID: "b3dc61e7-e5c5-48b2-8c88-84e1672a1858"). InnerVolumeSpecName "kube-api-access-554bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:02.668421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.668338    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:20:02.668421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:02.668418    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554bh\" (UniqueName: \"kubernetes.io/projected/b3dc61e7-e5c5-48b2-8c88-84e1672a1858-kube-api-access-554bh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:20:03.201386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.201315    4492 generic.go:356] "Generic (PLEG): container finished" podID="b3dc61e7-e5c5-48b2-8c88-84e1672a1858" containerID="5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61" exitCode=0
Nov 02 14:20:03.201386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.201380    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z" event={"ID":"b3dc61e7-e5c5-48b2-8c88-84e1672a1858","Type":"ContainerDied","Data":"5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61"}
Nov 02 14:20:03.201924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.201444    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z" event={"ID":"b3dc61e7-e5c5-48b2-8c88-84e1672a1858","Type":"ContainerDied","Data":"47de42d2a7529d7a2f92aea43c09fff96c45f2b726ec4c496af9b09ffa129a8b"}
Nov 02 14:20:03.201924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.201396    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"
Nov 02 14:20:03.201924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.201474    4492 scope.go:122] "RemoveContainer" containerID="5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61"
Nov 02 14:20:03.212416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.212397    4492 scope.go:122] "RemoveContainer" containerID="5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61"
Nov 02 14:20:03.212702 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:20:03.212680    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61\": container with ID starting with 5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61 not found: ID does not exist" containerID="5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61"
Nov 02 14:20:03.212748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.212710    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61"} err="failed to get container status \"5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61\": rpc error: code = NotFound desc = could not find container \"5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61\": container with ID starting with 5f74cee43a35c526c456095b197297c708136f4fc54e39a8b26f0360a3250a61 not found: ID does not exist"
Nov 02 14:20:03.249558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.249527    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"]
Nov 02 14:20:03.261106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.261070    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bb4ef/payload-processing-external-model-e2e-extmod-b-2bb4ef-5864xgs8z"]
Nov 02 14:20:03.583542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.583456    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dc61e7-e5c5-48b2-8c88-84e1672a1858" path="/var/lib/kubelet/pods/b3dc61e7-e5c5-48b2-8c88-84e1672a1858/volumes"
Nov 02 14:20:03.680762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:03.680729    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:05.379924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:05.379873    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:07.216864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:07.216815    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:07.520851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:07.520758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:09.056536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:09.056500    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:09.975979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:09.975919    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:10.093860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:10.093821    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:10.559883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:10.559818    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:10.709985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:10.709947    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:11.134453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:11.134422    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:12.376611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:12.376573    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:12.945021 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:12.944975    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:14.735065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:14.735023    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:16.519929 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:20:16.519890    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:20:16.624705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:16.624655    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:17.105451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:17.105413    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:17.245487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:17.245454    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:17.385321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:17.385235    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:18.429450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:18.429415    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:20.131736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:20.131707    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:21.933235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:21.933201    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:22.670026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:22.669994    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:23.656573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:23.656538    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:23.795392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:23.792973    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:25.528705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:25.528674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:27.241655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:27.241569    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:28.948460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:28.948428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:30.643992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:30.643961    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:32.427079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:32.427047    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:34.227085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:34.227047    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:35.994787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:35.994753    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:36.215428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:36.215377    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:36.340249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:36.340122    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:36.956542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:36.956503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:37.923537 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:37.923502    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:39.725647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:39.725608    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:41.536001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:41.535963    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:43.426570 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:43.426538    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:43.859865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:43.859788    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:44.010249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:44.010222    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:44.195154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:44.195122    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:44.721924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:44.721886    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:45.199136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:45.199092    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:45.327566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:45.327532    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:46.522097 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:20:46.522060    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:20:47.426655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:47.426620    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:48.082830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:48.082784    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:48.216983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:48.216942    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:49.232403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:49.232325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:49.809444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:49.809409    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:49.922873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:49.922840    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:51.106405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:51.106370    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:52.825655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:52.825625    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:54.624489 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:54.624459    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:56.567693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:56.567659    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:58.422167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:58.422121    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:20:58.531575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:20:58.531543    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:00.584840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.584803    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:00.757643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.757594    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"]
Nov 02 14:21:00.758751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.758724    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b3dc61e7-e5c5-48b2-8c88-84e1672a1858" containerName="payload-processing"
Nov 02 14:21:00.758751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.758749    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dc61e7-e5c5-48b2-8c88-84e1672a1858" containerName="payload-processing"
Nov 02 14:21:00.758969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.758955    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dc61e7-e5c5-48b2-8c88-84e1672a1858" containerName="payload-processing"
Nov 02 14:21:00.763590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.763566    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.780558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.780529    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"]
Nov 02 14:21:00.786280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.786038    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-598f75ee016d\""
Nov 02 14:21:00.786280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.786195    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-e2e--d8f5123f-dockercfg-qlpzh\""
Nov 02 14:21:00.786280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.786265    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ba664\""
Nov 02 14:21:00.787237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.786397    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-b8b0c81894ae\""
Nov 02 14:21:00.792650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.792621    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"openshift-service-ca.crt\""
Nov 02 14:21:00.792827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.792678    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"routing-overlay\""
Nov 02 14:21:00.792827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.792749    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"kube-root-ca.crt\""
Nov 02 14:21:00.805941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.805899    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdm82\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.806117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.806099    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.806179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.806154    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.806239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.806224    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.907262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.907164    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.907262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.907224    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.907262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.907245    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.907667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.907417    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdm82\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.908022 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.907996    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.908098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.908078    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.910038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.910015    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:00.929002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:00.928963    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdm82\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:01.080498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:01.080458    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:01.395547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:01.395517    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"]
Nov 02 14:21:01.397814 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:21:01.397783    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6078990e_2c23_493a_9edb_14ef3398de28.slice/crio-8106e121ca17a551911914b6047ae25ba068c9aa46cfdfd1187c738b9770518a WatchSource:0}: Error finding container 8106e121ca17a551911914b6047ae25ba068c9aa46cfdfd1187c738b9770518a: Status 404 returned error can't find the container with id 8106e121ca17a551911914b6047ae25ba068c9aa46cfdfd1187c738b9770518a
Nov 02 14:21:01.920031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:01.919959    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd" event={"ID":"6078990e-2c23-493a-9edb-14ef3398de28","Type":"ContainerStarted","Data":"21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e"}
Nov 02 14:21:01.920031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:01.920020    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd" event={"ID":"6078990e-2c23-493a-9edb-14ef3398de28","Type":"ContainerStarted","Data":"8106e121ca17a551911914b6047ae25ba068c9aa46cfdfd1187c738b9770518a"}
Nov 02 14:21:01.920628 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:01.920107    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:01.958569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:01.958514    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd" podStartSLOduration=1.95849831 podStartE2EDuration="1.95849831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:01.957426599 +0000 UTC m=+1426.501142193" watchObservedRunningTime="2026-11-02 14:21:01.95849831 +0000 UTC m=+1426.502213929"
Nov 02 14:21:02.354242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:02.354205    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:03.832230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:03.832198    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:04.224108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:04.224040    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:06.024306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:06.024260    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:07.643667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:07.643620    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:07.787903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:07.787867    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:07.900323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:07.900230    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:08.117369 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:08.117330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:09.620211 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:09.620175    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:10.847562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:10.847520    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:11.445305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:11.445272    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:12.489238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:12.489208    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:12.931342 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:12.931310    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:21:13.258455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:13.258336    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:16.001190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:15.976955    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:16.312340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:16.312246    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:16.625429 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:21:16.622855    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:21:17.839828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:17.839798    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:19.585681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:19.585641    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:21.351603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:21.351568    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:22.784376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:22.784328    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:23.145963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:23.145845    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:24.933343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:24.933312    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:26.789654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:26.789617    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:28.666172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:28.666133    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:30.576295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:30.576249    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:32.544236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:32.544205    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:34.436315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:34.436279    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:36.328414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:36.328346    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:38.230450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:38.230417    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:40.495381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:40.488963    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:43.188715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:43.188677    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:45.065475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:45.065440    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:45.507267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:45.507235    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:45.651379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:45.648252    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:46.518296 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:21:46.518259    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:21:47.045629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:47.045597    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:48.813550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:48.813522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:50.527089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:50.527053    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:52.294418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:52.294381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:53.889456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:53.889415    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:54.516582 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:54.516516    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:54.635026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:54.634987    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:54.762379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:54.754585    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:54.869637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:54.869560    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:55.558823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:55.558778    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:55.676162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:55.676131    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:55.868514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:55.868428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:56.121112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:56.121015    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:56.255625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:56.255533    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:56.470069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:56.470024    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:56.579666 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:56.579634    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:57.046304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:57.046260    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:57.787966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:57.787934    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:57.896649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:57.896620    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:59.458645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:59.458611    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:59.639158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:59.639124    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:21:59.751785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:21:59.751698    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:01.362648 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:01.362601    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:01.481804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:01.481769    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:01.607319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:01.607283    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:03.398683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:03.398641    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:04.776407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:04.776343    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:04.924613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:04.924573    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:05.160636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:05.160550    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:06.917668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:06.917633    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:07.771929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:07.771891    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:08.311534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:08.311502    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:08.418941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:08.418903    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:08.878392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:08.872832    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:09.177446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:09.177415    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:10.732346 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:10.732295    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:11.160451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:11.160246    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:11.273891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:11.273844    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:12.466222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:12.466183    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"]
Nov 02 14:22:12.619947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:12.619918    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:12.826543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:12.826458    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:15.256380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:15.256011    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:16.596234 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:16.596158    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:22:17.379612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.379576    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:17.466902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.466829    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd" podUID="6078990e-2c23-493a-9edb-14ef3398de28" containerName="payload-processing" containerID="cri-o://21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e" gracePeriod=25
Nov 02 14:22:17.529302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.529269    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:17.659655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.659527    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:17.777982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.777946    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:17.781973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.781950    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:22:17.870411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.870377    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82\") pod \"6078990e-2c23-493a-9edb-14ef3398de28\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") "
Nov 02 14:22:17.870636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.870524    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume\") pod \"6078990e-2c23-493a-9edb-14ef3398de28\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") "
Nov 02 14:22:17.870721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.870668    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume\") pod \"6078990e-2c23-493a-9edb-14ef3398de28\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") "
Nov 02 14:22:17.871748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.871006    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials\") pod \"6078990e-2c23-493a-9edb-14ef3398de28\" (UID: \"6078990e-2c23-493a-9edb-14ef3398de28\") "
Nov 02 14:22:17.871748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.871063    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume" pod "6078990e-2c23-493a-9edb-14ef3398de28" (UID: "6078990e-2c23-493a-9edb-14ef3398de28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:22:17.871748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.871407    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume" pod "6078990e-2c23-493a-9edb-14ef3398de28" (UID: "6078990e-2c23-493a-9edb-14ef3398de28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:22:17.871748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.871744    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:17.872112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.871766    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6078990e-2c23-493a-9edb-14ef3398de28-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:17.874278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.874252    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials" pod "6078990e-2c23-493a-9edb-14ef3398de28" (UID: "6078990e-2c23-493a-9edb-14ef3398de28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:22:17.874473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:17.874435    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82" pod "6078990e-2c23-493a-9edb-14ef3398de28" (UID: "6078990e-2c23-493a-9edb-14ef3398de28"). InnerVolumeSpecName "kube-api-access-zdm82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:22:18.041379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.040442    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:18.041379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.040473    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdm82\" (UniqueName: \"kubernetes.io/projected/6078990e-2c23-493a-9edb-14ef3398de28-kube-api-access-zdm82\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:18.244593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.244554    4492 generic.go:356] "Generic (PLEG): container finished" podID="6078990e-2c23-493a-9edb-14ef3398de28" containerID="21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e" exitCode=0
Nov 02 14:22:18.244802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.244610    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"
Nov 02 14:22:18.244802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.244738    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd" event={"ID":"6078990e-2c23-493a-9edb-14ef3398de28","Type":"ContainerDied","Data":"21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e"}
Nov 02 14:22:18.244802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.244784    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd" event={"ID":"6078990e-2c23-493a-9edb-14ef3398de28","Type":"ContainerDied","Data":"8106e121ca17a551911914b6047ae25ba068c9aa46cfdfd1187c738b9770518a"}
Nov 02 14:22:18.244998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.244813    4492 scope.go:122] "RemoveContainer" containerID="21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e"
Nov 02 14:22:18.262739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.262713    4492 scope.go:122] "RemoveContainer" containerID="21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e"
Nov 02 14:22:18.263120 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:18.263092    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e\": container with ID starting with 21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e not found: ID does not exist" containerID="21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e"
Nov 02 14:22:18.263217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.263132    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e"} err="failed to get container status \"21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e\": rpc error: code = NotFound desc = could not find container \"21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e\": container with ID starting with 21edff491bf3f2c7fd4e9c5aceb6ae8980137132aa196ed31625ec3e95b2195e not found: ID does not exist"
Nov 02 14:22:18.294610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.294512    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"]
Nov 02 14:22:18.298682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:18.298654    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc9qq9nd"]
Nov 02 14:22:19.335528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:19.335492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:19.587839 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:19.587757    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6078990e-2c23-493a-9edb-14ef3398de28" path="/var/lib/kubelet/pods/6078990e-2c23-493a-9edb-14ef3398de28/volumes"
Nov 02 14:22:22.207541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:22.207508    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:22.319250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:22.319217    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:22.634171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:22.634078    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:22.746046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:22.746012    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:22.900418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:22.900317    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:23.184180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:23.184144    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:23.295631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:23.295584    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:25.032343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:25.032312    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:26.833383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:26.831851    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:27.066414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:27.066372    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:27.451042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:27.451008    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:27.570601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:27.570553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:27.793331 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:27.793244    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:28.767629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:28.767591    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:29.379721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:29.379688    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:29.519548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:29.519517    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:30.640869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.640832    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:30.764854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.764820    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:30.884295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.884255    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:30.900308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.900235    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"]
Nov 02 14:22:30.900991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.900977    4492 cpu_manager.go:414] "containerMap: removing container" podUID="6078990e-2c23-493a-9edb-14ef3398de28" containerName="payload-processing"
Nov 02 14:22:30.901043 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.900993    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6078990e-2c23-493a-9edb-14ef3398de28" containerName="payload-processing"
Nov 02 14:22:30.901163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.901153    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="6078990e-2c23-493a-9edb-14ef3398de28" containerName="payload-processing"
Nov 02 14:22:30.908418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.908396    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:30.923724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.923692    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"]
Nov 02 14:22:30.927450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.927420    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 14:22:30.927590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.927549    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 14:22:30.927733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.927714    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 14:22:30.927785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.927768    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 14:22:30.927821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.927810    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-de08474d6f06\""
Nov 02 14:22:30.933744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.933715    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r4vzz\""
Nov 02 14:22:30.996443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:30.996407    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:31.091340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.091284    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.091695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.091670    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.091920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.091903    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.092076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.092062    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699pv\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.169345 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.169242    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"]
Nov 02 14:22:31.179099 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.179056    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-699pv plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr" podUID="e35d9ed2-2c73-47c0-bfa8-84ad53313617"
Nov 02 14:22:31.206330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.206287    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699pv\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.206902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.206874    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.252917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.252880    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.271808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.271769    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.272005 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.222519    4492 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 14:22:31.272076 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.272053    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume podName:e35d9ed2-2c73-47c0-bfa8-84ad53313617 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:31.77202611 +0000 UTC m=+1516.315741699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume") pod "payload-processing-external-model-7c999f66-ks4pr" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617") : configmap "payload-processing-external-model-plugins" not found
Nov 02 14:22:31.272076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.225268    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:31.272266 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.272246    4492 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de08474d6f06: secret "agc-cred-de08474d6f06" not found
Nov 02 14:22:31.272397 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.272268    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c999f66-ks4pr: secret "agc-cred-de08474d6f06" not found
Nov 02 14:22:31.272397 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.272325    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials podName:e35d9ed2-2c73-47c0-bfa8-84ad53313617 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:31.772309597 +0000 UTC m=+1516.316025181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials") pod "payload-processing-external-model-7c999f66-ks4pr" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617") : secret "agc-cred-de08474d6f06" not found
Nov 02 14:22:31.272397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.254570    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.272397 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.254831    4492 projected.go:196] Error preparing data for projected volume kube-api-access-699pv for pod models-as-a-service/payload-processing-external-model-7c999f66-ks4pr: failed to fetch token: serviceaccounts "payload-processing-external-model" not found
Nov 02 14:22:31.272606 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.272411    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv podName:e35d9ed2-2c73-47c0-bfa8-84ad53313617 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:31.772400287 +0000 UTC m=+1516.316115859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-699pv" (UniqueName: "kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv") pod "payload-processing-external-model-7c999f66-ks4pr" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found
Nov 02 14:22:31.411089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.411055    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.419123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.419096    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.474978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.474940    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume\") pod \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") "
Nov 02 14:22:31.475335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.475309    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume" pod "e35d9ed2-2c73-47c0-bfa8-84ad53313617" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:22:31.475666 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.475646    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.808636    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.808838    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:31.808924    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699pv\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv\") pod \"payload-processing-external-model-7c999f66-ks4pr\" (UID: \"e35d9ed2-2c73-47c0-bfa8-84ad53313617\") " pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.809502    4492 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.809568    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume podName:e35d9ed2-2c73-47c0-bfa8-84ad53313617 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:32.809546907 +0000 UTC m=+1517.353262494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume") pod "payload-processing-external-model-7c999f66-ks4pr" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617") : configmap "payload-processing-external-model-plugins" not found
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.810051    4492 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de08474d6f06: secret "agc-cred-de08474d6f06" not found
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.810066    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c999f66-ks4pr: secret "agc-cred-de08474d6f06" not found
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.810110    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials podName:e35d9ed2-2c73-47c0-bfa8-84ad53313617 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:32.810095419 +0000 UTC m=+1517.353811005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials") pod "payload-processing-external-model-7c999f66-ks4pr" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617") : secret "agc-cred-de08474d6f06" not found
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.823001    4492 projected.go:196] Error preparing data for projected volume kube-api-access-699pv for pod models-as-a-service/payload-processing-external-model-7c999f66-ks4pr: failed to fetch token: serviceaccounts "payload-processing-external-model" not found
Nov 02 14:22:31.832751 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:31.823074    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv podName:e35d9ed2-2c73-47c0-bfa8-84ad53313617 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:32.823054418 +0000 UTC m=+1517.366770005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-699pv" (UniqueName: "kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv") pod "payload-processing-external-model-7c999f66-ks4pr" (UID: "e35d9ed2-2c73-47c0-bfa8-84ad53313617") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found
Nov 02 14:22:32.418027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.417996    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"
Nov 02 14:22:32.491454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.491413    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"]
Nov 02 14:22:32.500836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.500797    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c999f66-ks4pr"]
Nov 02 14:22:32.622229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.622196    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699pv\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-kube-api-access-699pv\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:32.622229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.622221    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35d9ed2-2c73-47c0-bfa8-84ad53313617-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:32.622229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.622231    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e35d9ed2-2c73-47c0-bfa8-84ad53313617-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:22:32.723809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:32.723781    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:33.582821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:33.582780    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35d9ed2-2c73-47c0-bfa8-84ad53313617" path="/var/lib/kubelet/pods/e35d9ed2-2c73-47c0-bfa8-84ad53313617/volumes"
Nov 02 14:22:34.421380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:34.417757    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:34.556380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:34.554066    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:34.680249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:34.680210    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:35.144909 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:35.144783    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:35.308084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:35.308038    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:38.477472 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:38.477435    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:40.313396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:40.313320    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:42.079990 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:42.079956    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:43.792908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:43.792875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:44.236454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:44.236416    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:45.613191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:45.613159    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:46.521844 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:22:46.521807    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:22:47.304518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:47.304480    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:47.422489 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:47.422455    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:47.792389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:47.792338    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:47.997925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:47.997880    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:49.394332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:49.394304    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:51.445541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:51.445505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:52.118162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:52.118126    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:52.233134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:52.233100    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:53.423165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:53.423131    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:53.872657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:53.872622    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:53.981566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:53.981534    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:55.228326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:55.228292    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:57.022116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:57.022076    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:22:58.939642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:22:58.939332    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:00.844486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:00.844438    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:01.276873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:01.276839    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:01.408449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:01.408413    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:01.542842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:01.542746    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:02.815658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:02.815628    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:03.538087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:03.538047    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:03.651227 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:03.651195    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:04.795490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:04.795460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:04.965970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:04.965932    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:06.585254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:06.585222    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:07.873820 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:07.873789    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:07.986894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:07.986853    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:08.190988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:08.190961    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:08.685622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:08.685585    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:08.794470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:08.794377    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:08.904796 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:08.904748    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:09.037893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:09.037864    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:09.741683 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:09.741653    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:09.826001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:09.825941    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"]
Nov 02 14:23:09.826304 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:09.826272    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy" containerID="cri-o://7b24b865c628c308390e58ce2b12243b12abdb9a826d46af4050ea2cd4c9cd6b" gracePeriod=30
Nov 02 14:23:09.834412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:09.834392    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"]
Nov 02 14:23:09.834615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:09.834593    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy" containerID="cri-o://8b1cc1b9fe51a17827b759e83e20ee3b3bc8b9f310243340481a243b068cfd0a" gracePeriod=30
Nov 02 14:23:10.577403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:10.577374    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:10.695709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:10.695676    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:10.844999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:10.844298    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:12.681134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:12.681097    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:12.832385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:12.823199    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:13.140227 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:13.140090    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:13.275709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:13.275672    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:14.779476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:14.779443    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:14.960555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:14.960518    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:15.084136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.084033    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:15.086848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.086575    4492 generic.go:356] "Generic (PLEG): container finished" podID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerID="8b1cc1b9fe51a17827b759e83e20ee3b3bc8b9f310243340481a243b068cfd0a" exitCode=0
Nov 02 14:23:15.086848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.086764    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" event={"ID":"2aa13a40-bd00-490f-9c8a-a7a915a7198c","Type":"ContainerDied","Data":"8b1cc1b9fe51a17827b759e83e20ee3b3bc8b9f310243340481a243b068cfd0a"}
Nov 02 14:23:15.090707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.090442    4492 generic.go:356] "Generic (PLEG): container finished" podID="163c67ab-be95-4ae5-af35-1f68185385db" containerID="7b24b865c628c308390e58ce2b12243b12abdb9a826d46af4050ea2cd4c9cd6b" exitCode=0
Nov 02 14:23:15.090707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.090532    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" event={"ID":"163c67ab-be95-4ae5-af35-1f68185385db","Type":"ContainerDied","Data":"7b24b865c628c308390e58ce2b12243b12abdb9a826d46af4050ea2cd4c9cd6b"}
Nov 02 14:23:15.090707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.090588    4492 scope.go:122] "RemoveContainer" containerID="1c6773568c89387a3b8502e8d98c970e433826980f3466528ba7adf48ae4d975"
Nov 02 14:23:15.498707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.498670    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:23:15.521019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.520989    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:23:15.586230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586197    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.586442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586251    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.586442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586288    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.586442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586345    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.586625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586457    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.586625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586502    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.586728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586635    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.586728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586714    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.586847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586775    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.586847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586825    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.586950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586863    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.586950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586903    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.587061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.586957    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.587061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587030    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.587168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587074    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.587168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587115    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.587269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587172    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert\") pod \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\" (UID: \"2aa13a40-bd00-490f-9c8a-a7a915a7198c\") "
Nov 02 14:23:15.587269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587224    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token\") pod \"163c67ab-be95-4ae5-af35-1f68185385db\" (UID: \"163c67ab-be95-4ae5-af35-1f68185385db\") "
Nov 02 14:23:15.587705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587677    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.587952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.587933    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.588689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.588666    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:15.588932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.588679    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.589040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.588772    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.589132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.588954    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:15.589217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.589177    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.589927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.589895    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.591913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.591504    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:15.593646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.593377    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.594514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.594491    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.594738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.594711    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "kube-api-access-qbtht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:15.594932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.594910    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.598753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.595769    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "kube-api-access-hzjhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:15.598753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.595844    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:15.598753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.596210    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.598753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.598115    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token" pod "163c67ab-be95-4ae5-af35-1f68185385db" (UID: "163c67ab-be95-4ae5-af35-1f68185385db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:15.604534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.604500    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:15.604687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.604613    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token" pod "2aa13a40-bd00-490f-9c8a-a7a915a7198c" (UID: "2aa13a40-bd00-490f-9c8a-a7a915a7198c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:15.689169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689135    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689163    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163c67ab-be95-4ae5-af35-1f68185385db-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689173    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689193    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163c67ab-be95-4ae5-af35-1f68185385db-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689200    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689211    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689220    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbtht\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-kube-api-access-qbtht\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689228    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689236    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689243    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689250    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689257    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689264    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689271    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjhm\" (UniqueName: \"kubernetes.io/projected/163c67ab-be95-4ae5-af35-1f68185385db-kube-api-access-hzjhm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689279    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689288    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aa13a40-bd00-490f-9c8a-a7a915a7198c-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:15.689437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:15.689296    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163c67ab-be95-4ae5-af35-1f68185385db-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:16.110436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.110399    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"
Nov 02 14:23:16.110933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.110399    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg" event={"ID":"163c67ab-be95-4ae5-af35-1f68185385db","Type":"ContainerDied","Data":"fb39b4d1ebfcf6e46275d7d41abe8bb70a218884eb15e7ec141443575b706b80"}
Nov 02 14:23:16.110933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.110556    4492 scope.go:122] "RemoveContainer" containerID="7b24b865c628c308390e58ce2b12243b12abdb9a826d46af4050ea2cd4c9cd6b"
Nov 02 14:23:16.112910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.112888    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"
Nov 02 14:23:16.112910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.112894    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv" event={"ID":"2aa13a40-bd00-490f-9c8a-a7a915a7198c","Type":"ContainerDied","Data":"9cc110fde97d7a933cc53aa1bf9ef78a589380eb11ec3f53f4c754c6fca98426"}
Nov 02 14:23:16.123250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.123232    4492 scope.go:122] "RemoveContainer" containerID="8b1cc1b9fe51a17827b759e83e20ee3b3bc8b9f310243340481a243b068cfd0a"
Nov 02 14:23:16.167780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.167750    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"]
Nov 02 14:23:16.177282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.177255    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-hwlcv"]
Nov 02 14:23:16.203431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.203396    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"]
Nov 02 14:23:16.212389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.212343    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-zbsqg"]
Nov 02 14:23:16.502774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:16.502730    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:16.518713 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:23:16.518655    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:23:17.592043 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:17.592011    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163c67ab-be95-4ae5-af35-1f68185385db" path="/var/lib/kubelet/pods/163c67ab-be95-4ae5-af35-1f68185385db/volumes"
Nov 02 14:23:17.592740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:17.592723    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" path="/var/lib/kubelet/pods/2aa13a40-bd00-490f-9c8a-a7a915a7198c/volumes"
Nov 02 14:23:18.215982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:18.215946    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:20.026048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:20.026014    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:22.371092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:22.371052    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:23.107023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:23.106979    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:23.536044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:23.535999    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:24.411804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:24.411733    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:26.093290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:26.093225    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:27.837485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:27.837428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:29.506149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:29.506084    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:31.348456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:31.348413    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:33.041749 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:33.041709    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:34.622794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.622756    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"]
Nov 02 14:23:34.623694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623671    4492 cpu_manager.go:414] "containerMap: removing container" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy"
Nov 02 14:23:34.623694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623693    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy"
Nov 02 14:23:34.623874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623709    4492 cpu_manager.go:414] "containerMap: removing container" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy"
Nov 02 14:23:34.623874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623716    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy"
Nov 02 14:23:34.623874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623767    4492 cpu_manager.go:414] "containerMap: removing container" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy"
Nov 02 14:23:34.623874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623775    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy"
Nov 02 14:23:34.624079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623948    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy"
Nov 02 14:23:34.624079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623961    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="163c67ab-be95-4ae5-af35-1f68185385db" containerName="istio-proxy"
Nov 02 14:23:34.624079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.623978    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa13a40-bd00-490f-9c8a-a7a915a7198c" containerName="istio-proxy"
Nov 02 14:23:34.629300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.629279    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.659527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.659493    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8416e2e2-serving-cert\""
Nov 02 14:23:34.660576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.660545    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"]
Nov 02 14:23:34.671371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.671316    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.671565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.671538    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.671658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.671603    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27gg\" (UniqueName: \"kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.671719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.671702    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.729796 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.729767    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:34.773048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.773015    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.773242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.773118    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.773242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.773152    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27gg\" (UniqueName: \"kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.773242 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:23:34.773173    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-serving-cert: secret "maas-api-e2e-ait-8416e2e2-serving-cert" not found
Nov 02 14:23:34.773242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.773224    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.773242 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:23:34.773242    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls podName:4e8ad411-9b60-47aa-9d32-a39208790378 nodeName:}" failed. No retries permitted until 2026-11-02 14:23:35.273221683 +0000 UTC m=+1579.816937258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls") pod "maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" (UID: "4e8ad411-9b60-47aa-9d32-a39208790378") : secret "maas-api-e2e-ait-8416e2e2-serving-cert" not found
Nov 02 14:23:34.773933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.773901    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.776571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.776550    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.800975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.800941    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27gg\" (UniqueName: \"kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:34.857014 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.856973    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:34.968806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:34.968774    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:35.286853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:35.286759    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:35.289548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:35.289517    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\") pod \"maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:35.546771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:35.546682    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:35.949279 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:35.944829    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"]
Nov 02 14:23:35.956745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:35.956720    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:23:36.306379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:36.306279    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" event={"ID":"4e8ad411-9b60-47aa-9d32-a39208790378","Type":"ContainerStarted","Data":"01d986ed79b1881dbb21d88ee30a1542456cdebe8883a4bdf11c7f29e2a6ec38"}
Nov 02 14:23:36.539244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:36.539208    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:37.315307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:37.315267    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" event={"ID":"4e8ad411-9b60-47aa-9d32-a39208790378","Type":"ContainerStarted","Data":"cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0"}
Nov 02 14:23:37.315751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:37.315329    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:37.354412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:37.354340    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" podStartSLOduration=2.967561045 podStartE2EDuration="3.354322505s" totalImagesPullingTime="386.76146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:37.353494371 +0000 UTC m=+1581.897209967" watchObservedRunningTime="2026-11-02 14:23:37.354322505 +0000 UTC m=+1581.898038099"
Nov 02 14:23:38.408257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:38.408221    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:40.146927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:40.146891    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"]
Nov 02 14:23:40.147332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:40.147094    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy" containerID="cri-o://ccef464c7f47addc54e9a87d0daa5a83902a760310927eee7f281d4e3f0c5528" gracePeriod=30
Nov 02 14:23:40.357831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:40.357798    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:42.885078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:42.885028    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:43.017249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:43.017187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:43.364266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:43.364231    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:23:44.349703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:44.349669    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:44.921527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:44.921486    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:45.144389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.144054    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:45.447276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.447244    4492 generic.go:356] "Generic (PLEG): container finished" podID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerID="ccef464c7f47addc54e9a87d0daa5a83902a760310927eee7f281d4e3f0c5528" exitCode=0
Nov 02 14:23:45.447767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.447322    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" event={"ID":"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9","Type":"ContainerDied","Data":"ccef464c7f47addc54e9a87d0daa5a83902a760310927eee7f281d4e3f0c5528"}
Nov 02 14:23:45.550454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.550190    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:45.611169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.610978    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:23:45.811139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.811020    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.811687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.811564    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.812009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.811987    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.812192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.812175    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.812385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.812368    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.812638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.812611    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.812814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.812797    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.813015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.812999    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.813173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.813158    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token\") pod \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\" (UID: \"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9\") "
Nov 02 14:23:45.814848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.814824    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:45.815273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.815255    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:45.817109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.817071    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:45.817293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.817260    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:45.822102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.821116    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:45.823927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.822951    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "kube-api-access-hc4hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:45.830497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.829859    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:45.831802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.831727    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:45.832113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.832068    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo" pod "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" (UID: "0f7ca76f-2ea1-491b-bd7c-58b330e35ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:45.916312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916253    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916308    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916323    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916338    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4hx\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-kube-api-access-hc4hx\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916370    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916388    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916401    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916415    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:45.916613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:45.916427    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:23:46.458022 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:46.457994    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"
Nov 02 14:23:46.458518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:46.458028    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4" event={"ID":"0f7ca76f-2ea1-491b-bd7c-58b330e35ab9","Type":"ContainerDied","Data":"f443b2f8c662410385449ffd1d3e5010255b0b85524bd7ada6e256b0b48a0dc0"}
Nov 02 14:23:46.458518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:46.458103    4492 scope.go:122] "RemoveContainer" containerID="ccef464c7f47addc54e9a87d0daa5a83902a760310927eee7f281d4e3f0c5528"
Nov 02 14:23:46.516897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:46.516863    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"]
Nov 02 14:23:46.531319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:46.531270    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-5k5q4"]
Nov 02 14:23:46.537664 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:23:46.537612    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:23:47.583571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:47.583539    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" path="/var/lib/kubelet/pods/0f7ca76f-2ea1-491b-bd7c-58b330e35ab9/volumes"
Nov 02 14:23:47.704664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:47.704622    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:48.608208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:48.608171    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:48.724744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:48.724707    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:49.746089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:49.746059    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:50.410303 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.410242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:50.523555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.523521    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:50.585634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.585600    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"]
Nov 02 14:23:50.586380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.586349    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy"
Nov 02 14:23:50.586466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.586381    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy"
Nov 02 14:23:50.586522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.586490    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7ca76f-2ea1-491b-bd7c-58b330e35ab9" containerName="istio-proxy"
Nov 02 14:23:50.591751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.591729    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:50.611073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.611039    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"]
Nov 02 14:23:50.615195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.615164    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-673ba8\""
Nov 02 14:23:50.615441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.615259    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-673ba8-dockercfg-6p8fh\""
Nov 02 14:23:50.696381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.678437    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"]
Nov 02 14:23:50.732388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.723387    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:50.732878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.732842    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx25g\" (UniqueName: \"kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g\") pod \"payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:50.836376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.815034    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"]
Nov 02 14:23:50.854802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.828512    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:50.929276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.929226    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx25g\" (UniqueName: \"kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g\") pod \"payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:50.929923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.929888    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:50.931169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.931121    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:50.954372 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:50.954274    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx25g\" (UniqueName: \"kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g\") pod \"payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:51.031728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.031695    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhs9r\" (UniqueName: \"kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r\") pod \"payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.031903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.031766    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.133123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.133085    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhs9r\" (UniqueName: \"kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r\") pod \"payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.133325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.133174    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.134080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.134058    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.154888 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.154857    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhs9r\" (UniqueName: \"kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r\") pod \"payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.208176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.208086    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:51.218681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.218628    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:51.551238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.551211    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"]
Nov 02 14:23:51.553976 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:23:51.553931    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d9f9ec_f170_4d5e_973c_7e03510570ab.slice/crio-f6bf1e33322e4008a28ac1cea057968a94c8d9d8d930287c80e00448c1063346 WatchSource:0}: Error finding container f6bf1e33322e4008a28ac1cea057968a94c8d9d8d930287c80e00448c1063346: Status 404 returned error can't find the container with id f6bf1e33322e4008a28ac1cea057968a94c8d9d8d930287c80e00448c1063346
Nov 02 14:23:51.565462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.565435    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"]
Nov 02 14:23:51.571529 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:23:51.571496    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab230828_8405_43cf_82c9_ff18683d733e.slice/crio-608f1613572fccaa2287243c3cf3909b3a689f81c4bdeca071e2fbe5af5afdd5 WatchSource:0}: Error finding container 608f1613572fccaa2287243c3cf3909b3a689f81c4bdeca071e2fbe5af5afdd5: Status 404 returned error can't find the container with id 608f1613572fccaa2287243c3cf3909b3a689f81c4bdeca071e2fbe5af5afdd5
Nov 02 14:23:51.633677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:51.633643    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:52.510121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.510088    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb" event={"ID":"ab230828-8405-43cf-82c9-ff18683d733e","Type":"ContainerStarted","Data":"7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8"}
Nov 02 14:23:52.510569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.510155    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb" event={"ID":"ab230828-8405-43cf-82c9-ff18683d733e","Type":"ContainerStarted","Data":"608f1613572fccaa2287243c3cf3909b3a689f81c4bdeca071e2fbe5af5afdd5"}
Nov 02 14:23:52.510569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.510287    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:23:52.511826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.511802    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk" event={"ID":"e1d9f9ec-f170-4d5e-973c-7e03510570ab","Type":"ContainerStarted","Data":"288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78"}
Nov 02 14:23:52.511914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.511850    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk" event={"ID":"e1d9f9ec-f170-4d5e-973c-7e03510570ab","Type":"ContainerStarted","Data":"f6bf1e33322e4008a28ac1cea057968a94c8d9d8d930287c80e00448c1063346"}
Nov 02 14:23:52.511914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.511880    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:23:52.549634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.549580    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb" podStartSLOduration=2.549563517 podStartE2EDuration="2.549563517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:52.546035024 +0000 UTC m=+1597.089750619" watchObservedRunningTime="2026-11-02 14:23:52.549563517 +0000 UTC m=+1597.093279108"
Nov 02 14:23:52.590779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:52.590722    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk" podStartSLOduration=2.590707771 podStartE2EDuration="2.590707771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:52.586295062 +0000 UTC m=+1597.130010657" watchObservedRunningTime="2026-11-02 14:23:52.590707771 +0000 UTC m=+1597.134423364"
Nov 02 14:23:53.441528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:53.441491    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:55.238993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:55.238951    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:57.363988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:57.363946    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:23:59.992059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:23:59.992005    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:01.716466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:01.716422    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:03.557639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:03.557609    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:24:03.565815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:03.565795    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:24:03.591375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:03.591334    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:05.429467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:05.429428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:07.347137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:07.347108    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:08.014207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:08.014178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:09.139268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:09.139236    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:09.928524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:09.928492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:10.039179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:10.039139    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:10.938760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:10.938718    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:12.578811 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.578771    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"]
Nov 02 14:24:12.583615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.583591    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.608554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.608512    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-673ba8-serving-cert\""
Nov 02 14:24:12.613061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.613032    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"]
Nov 02 14:24:12.669665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.669630    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:12.691765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.691736    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.691942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.691812    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.692011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.691992    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.692056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.692028    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnxlx\" (UniqueName: \"kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.782630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.782586    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:12.793015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.792986    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.793195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.793175    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.793239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.793218    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnxlx\" (UniqueName: \"kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.793409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.793388    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.793933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.793896    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.796005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.795980    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.796383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.796339    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.814558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.814532    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnxlx\" (UniqueName: \"kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx\") pod \"maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:12.901451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.901230    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:12.901864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:12.901841    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:13.273456 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:24:13.273421    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae0bf9a_fd3a_4399_b69f_bef6f8f5b7f4.slice/crio-8c665ffcffeba1ad4fa701b9fa86bfd9c403c5e79d2eb33c925411f0216c87ad WatchSource:0}: Error finding container 8c665ffcffeba1ad4fa701b9fa86bfd9c403c5e79d2eb33c925411f0216c87ad: Status 404 returned error can't find the container with id 8c665ffcffeba1ad4fa701b9fa86bfd9c403c5e79d2eb33c925411f0216c87ad
Nov 02 14:24:13.274506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:13.274480    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"]
Nov 02 14:24:13.785428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:13.785390    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f" event={"ID":"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4","Type":"ContainerStarted","Data":"a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26"}
Nov 02 14:24:13.785832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:13.785456    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f" event={"ID":"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4","Type":"ContainerStarted","Data":"8c665ffcffeba1ad4fa701b9fa86bfd9c403c5e79d2eb33c925411f0216c87ad"}
Nov 02 14:24:13.785832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:13.785485    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:13.825746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:13.825698    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f" podStartSLOduration=1.480350335 podStartE2EDuration="1.825683688s" totalImagesPullingTime="345.333353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:13.823481136 +0000 UTC m=+1618.367196731" watchObservedRunningTime="2026-11-02 14:24:13.825683688 +0000 UTC m=+1618.369399281"
Nov 02 14:24:14.802679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:14.802641    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:16.012592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:16.012556    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:16.130400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:16.129219    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:16.530438 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:24:16.530393    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:24:16.621893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:16.621824    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:19.408379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:19.408167    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:19.798850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:19.798820    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:24:21.246371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:21.246323    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:23.024090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:23.024061    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:23.236323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:23.236295    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:24.715071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:24.715025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:26.470949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:26.470905    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:27.889347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:27.889309    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"]
Nov 02 14:24:27.889899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:27.889575    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" podUID="4e8ad411-9b60-47aa-9d32-a39208790378" containerName="maas-api" containerID="cri-o://cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0" gracePeriod=30
Nov 02 14:24:27.959309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:27.959281    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:28.312379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.297049    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:24:28.469680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.469588    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs\") pod \"4e8ad411-9b60-47aa-9d32-a39208790378\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") "
Nov 02 14:24:28.469680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.469652    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca\") pod \"4e8ad411-9b60-47aa-9d32-a39208790378\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") "
Nov 02 14:24:28.469929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.469687    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg\" (UniqueName: \"kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg\") pod \"4e8ad411-9b60-47aa-9d32-a39208790378\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") "
Nov 02 14:24:28.469929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.469730    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\") pod \"4e8ad411-9b60-47aa-9d32-a39208790378\" (UID: \"4e8ad411-9b60-47aa-9d32-a39208790378\") "
Nov 02 14:24:28.480206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.480141    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca" pod "4e8ad411-9b60-47aa-9d32-a39208790378" (UID: "4e8ad411-9b60-47aa-9d32-a39208790378"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:28.544409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.521530    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:28.544409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.523078    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg" pod "4e8ad411-9b60-47aa-9d32-a39208790378" (UID: "4e8ad411-9b60-47aa-9d32-a39208790378"). InnerVolumeSpecName "kube-api-access-s27gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:28.544409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.523254    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs" pod "4e8ad411-9b60-47aa-9d32-a39208790378" (UID: "4e8ad411-9b60-47aa-9d32-a39208790378"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:24:28.553382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.546595    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls" pod "4e8ad411-9b60-47aa-9d32-a39208790378" (UID: "4e8ad411-9b60-47aa-9d32-a39208790378"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:24:28.574837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.571850    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:24:28.574837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.571885    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e8ad411-9b60-47aa-9d32-a39208790378-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:24:28.574837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.571901    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s27gg\" (UniqueName: \"kubernetes.io/projected/4e8ad411-9b60-47aa-9d32-a39208790378-kube-api-access-s27gg\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:24:28.574837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.571915    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e8ad411-9b60-47aa-9d32-a39208790378-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:24:28.670670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.670635    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:28.796451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:28.794063    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:29.007344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.007310    4492 generic.go:356] "Generic (PLEG): container finished" podID="4e8ad411-9b60-47aa-9d32-a39208790378" containerID="cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0" exitCode=0
Nov 02 14:24:29.007762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.007381    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" event={"ID":"4e8ad411-9b60-47aa-9d32-a39208790378","Type":"ContainerDied","Data":"cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0"}
Nov 02 14:24:29.007762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.007412    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"
Nov 02 14:24:29.007762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.007456    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt" event={"ID":"4e8ad411-9b60-47aa-9d32-a39208790378","Type":"ContainerDied","Data":"01d986ed79b1881dbb21d88ee30a1542456cdebe8883a4bdf11c7f29e2a6ec38"}
Nov 02 14:24:29.007762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.007479    4492 scope.go:122] "RemoveContainer" containerID="cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0"
Nov 02 14:24:29.019892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.019875    4492 scope.go:122] "RemoveContainer" containerID="cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0"
Nov 02 14:24:29.020195 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:24:29.020175    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0\": container with ID starting with cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0 not found: ID does not exist" containerID="cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0"
Nov 02 14:24:29.020244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.020206    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0"} err="failed to get container status \"cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0\": rpc error: code = NotFound desc = could not find container \"cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0\": container with ID starting with cc8ceb178ed4451d27975c27a92e99b6c7fcb2bccbaaaa92dce832bc783bc6e0 not found: ID does not exist"
Nov 02 14:24:29.064631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.064551    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"]
Nov 02 14:24:29.074587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.074560    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8416e2e2-5ffc5c549c-zdvrt"]
Nov 02 14:24:29.581730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:29.581693    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8ad411-9b60-47aa-9d32-a39208790378" path="/var/lib/kubelet/pods/4e8ad411-9b60-47aa-9d32-a39208790378/volumes"
Nov 02 14:24:30.386091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:30.386058    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:30.560407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:30.548848    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:31.055099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:31.055049    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:31.567292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:31.567250    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:31.685424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:31.680619    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:32.065297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:32.065266    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:32.194911 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:32.194874    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:32.384333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:32.384251    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:34.309521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:34.309491    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:34.445414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:34.444924    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:34.584769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:34.584695    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:35.031954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:35.031914    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:35.152182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:35.152155    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:36.629469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:36.629439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:37.630574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:37.630512    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:37.810241 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:37.810208    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:38.568335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:38.568300    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:40.658837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:40.658792    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:42.649606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:42.649551    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:44.506111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:44.506073    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:44.696717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:44.696691    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:46.229293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:46.229257    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:46.520077 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:24:46.519985    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:24:48.103375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:48.103325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:48.520056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:48.520025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:49.967786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:49.967738    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:51.862967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:51.862895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:53.646473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:53.646440    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:55.120485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:55.120401    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:55.475330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:55.475289    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:55.916483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:55.916393    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:56.031818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:56.031790    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:56.220123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:56.220083    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:56.368799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:56.368763    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:56.479597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:56.479510    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:56.669706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:56.669675    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:57.313983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:57.313931    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:57.424718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:57.424684    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:57.897378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:57.893161    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"]
Nov 02 14:24:57.897378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:57.893467    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy" containerID="cri-o://9a4c2262132ccea878f41683fe1a8a8880be688403ffce8cd180c46de0012947" gracePeriod=30
Nov 02 14:24:58.848076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:58.848049    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:59.204494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:59.204460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:59.349888 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:59.349846    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:59.478394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:59.478284    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:24:59.875433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:24:59.869831    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:00.173290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:00.173206    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:00.771608 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:00.771561    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:00.951844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:00.951811    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:01.695374 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:01.695312    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:03.436181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.436147    4492 generic.go:356] "Generic (PLEG): container finished" podID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerID="9a4c2262132ccea878f41683fe1a8a8880be688403ffce8cd180c46de0012947" exitCode=0
Nov 02 14:25:03.436781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.436307    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" event={"ID":"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c","Type":"ContainerDied","Data":"9a4c2262132ccea878f41683fe1a8a8880be688403ffce8cd180c46de0012947"}
Nov 02 14:25:03.540902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.540870    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:25:03.631277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631240    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631295    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631411    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631445    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631475    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631506    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631561    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631707    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.631815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631727    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:03.631928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.631849    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket\") pod \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\" (UID: \"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c\") "
Nov 02 14:25:03.633018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.632720    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.633018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.632962    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:03.634142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.633562    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:03.634142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.634067    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:03.634142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.634093    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:25:03.634142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.634121    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:03.634377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.634265    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "kube-api-access-kwmvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:03.634473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.634452    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:03.636180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.636158    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token" pod "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" (UID: "0a44b758-5954-41b5-8f9a-4c0c6f24ad3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:03.734366 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734325    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmvx\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-kube-api-access-kwmvx\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734380    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734396    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734408    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734421    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734433    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734446    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:03.734552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:03.734488    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:04.135640 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:04.135610    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:04.445055 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:04.445026    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"
Nov 02 14:25:04.445493 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:04.445054    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb" event={"ID":"0a44b758-5954-41b5-8f9a-4c0c6f24ad3c","Type":"ContainerDied","Data":"2c5feeced79084c3c42485a251869e1b6a4a905a431d8d24988a19fb13228dd5"}
Nov 02 14:25:04.445493 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:04.445121    4492 scope.go:122] "RemoveContainer" containerID="9a4c2262132ccea878f41683fe1a8a8880be688403ffce8cd180c46de0012947"
Nov 02 14:25:04.504756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:04.504723    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"]
Nov 02 14:25:04.514164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:04.514130    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-w84bb"]
Nov 02 14:25:05.594029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:05.593990    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" path="/var/lib/kubelet/pods/0a44b758-5954-41b5-8f9a-4c0c6f24ad3c/volumes"
Nov 02 14:25:05.947477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:05.947444    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:07.731903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:07.731868    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:08.134042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:08.133935    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:09.617206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:09.617175    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:11.431757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:11.431714    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:13.035686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:13.035648    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:13.340590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:13.340302    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:15.144584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:15.144552    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:16.519027 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:25:16.518985    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:25:16.849912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:16.849827    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:18.568853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:18.568821    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:20.293401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.288923    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:20.405494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.405460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:20.665397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.665276    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"]
Nov 02 14:25:20.666052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.666034    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy"
Nov 02 14:25:20.666120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.666052    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy"
Nov 02 14:25:20.666120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.666076    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4e8ad411-9b60-47aa-9d32-a39208790378" containerName="maas-api"
Nov 02 14:25:20.666120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.666082    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8ad411-9b60-47aa-9d32-a39208790378" containerName="maas-api"
Nov 02 14:25:20.666280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.666181    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a44b758-5954-41b5-8f9a-4c0c6f24ad3c" containerName="istio-proxy"
Nov 02 14:25:20.666280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.666194    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8ad411-9b60-47aa-9d32-a39208790378" containerName="maas-api"
Nov 02 14:25:20.670983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.670961    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.685984 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.685265    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b98b233190d\""
Nov 02 14:25:20.691467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.691437    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"]
Nov 02 14:25:20.695560 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.695535    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 14:25:20.695877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.695578    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 14:25:20.695877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.695678    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-56zdx\""
Nov 02 14:25:20.696059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.695777    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 14:25:20.696059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.695829    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 14:25:20.729575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.729539    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.729785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.729611    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plt5t\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.729869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.729796    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.729869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.729846    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.831770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.831738    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.831976 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.831791    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plt5t\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.831976 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.831862    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.831976 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.831894    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.832673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.832621    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.832673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.832647    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.834556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.834532    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.853807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.853770    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plt5t\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t\") pod \"payload-processing-external-model-5c4dd54948-xjxbh\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") " pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:20.996915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:20.996624    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:21.436673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:21.436639    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"]
Nov 02 14:25:21.437112 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:25:21.437087    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5f7880_90a8_44cb_b8c1_191b5d68739d.slice/crio-20c8281d76cd082d9ae16af48ba2257294ff0bc9f6661a7d406d6f2f1a94055b WatchSource:0}: Error finding container 20c8281d76cd082d9ae16af48ba2257294ff0bc9f6661a7d406d6f2f1a94055b: Status 404 returned error can't find the container with id 20c8281d76cd082d9ae16af48ba2257294ff0bc9f6661a7d406d6f2f1a94055b
Nov 02 14:25:21.619546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:21.619510    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh" event={"ID":"ca5f7880-90a8-44cb-b8c1-191b5d68739d","Type":"ContainerStarted","Data":"5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5"}
Nov 02 14:25:21.619725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:21.619563    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh" event={"ID":"ca5f7880-90a8-44cb-b8c1-191b5d68739d","Type":"ContainerStarted","Data":"20c8281d76cd082d9ae16af48ba2257294ff0bc9f6661a7d406d6f2f1a94055b"}
Nov 02 14:25:21.619725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:21.619590    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:21.686433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:21.673386    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh" podStartSLOduration=1.673370098 podStartE2EDuration="1.673370098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:21.667427333 +0000 UTC m=+1686.211142926" watchObservedRunningTime="2026-11-02 14:25:21.673370098 +0000 UTC m=+1686.217085687"
Nov 02 14:25:22.233001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:22.232970    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:23.359392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:23.359341    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:24.036289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:24.036256    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:25.734534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:25.734498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:27.534313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:27.534261    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:29.712876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:29.712838    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:32.615545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:32.615505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:32.636828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:32.636798    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:33.166665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:33.166623    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"]
Nov 02 14:25:33.200658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:33.200615    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:33.317496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:33.317452    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:34.654680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:34.654642    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:36.442913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:36.442879    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:38.167848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.167789    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh" podUID="ca5f7880-90a8-44cb-b8c1-191b5d68739d" containerName="payload-processing" containerID="cri-o://5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5" gracePeriod=25
Nov 02 14:25:38.296578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.296548    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:38.529719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.529693    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:38.632087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.632050    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t\") pod \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") "
Nov 02 14:25:38.632292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.632256    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume\") pod \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") "
Nov 02 14:25:38.632401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.632384    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume\") pod \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") "
Nov 02 14:25:38.632485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.632457    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials\") pod \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\" (UID: \"ca5f7880-90a8-44cb-b8c1-191b5d68739d\") "
Nov 02 14:25:38.633527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.632903    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume" pod "ca5f7880-90a8-44cb-b8c1-191b5d68739d" (UID: "ca5f7880-90a8-44cb-b8c1-191b5d68739d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:38.633527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.633028    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume" pod "ca5f7880-90a8-44cb-b8c1-191b5d68739d" (UID: "ca5f7880-90a8-44cb-b8c1-191b5d68739d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:38.633527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.633311    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:38.633527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.633334    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f7880-90a8-44cb-b8c1-191b5d68739d-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:38.636427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.636347    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials" pod "ca5f7880-90a8-44cb-b8c1-191b5d68739d" (UID: "ca5f7880-90a8-44cb-b8c1-191b5d68739d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:38.636427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.636415    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t" pod "ca5f7880-90a8-44cb-b8c1-191b5d68739d" (UID: "ca5f7880-90a8-44cb-b8c1-191b5d68739d"). InnerVolumeSpecName "kube-api-access-plt5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:38.734795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.734757    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:38.734795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.734794    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plt5t\" (UniqueName: \"kubernetes.io/projected/ca5f7880-90a8-44cb-b8c1-191b5d68739d-kube-api-access-plt5t\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:25:38.802230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.802140    4492 generic.go:356] "Generic (PLEG): container finished" podID="ca5f7880-90a8-44cb-b8c1-191b5d68739d" containerID="5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5" exitCode=0
Nov 02 14:25:38.802230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.802187    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh" event={"ID":"ca5f7880-90a8-44cb-b8c1-191b5d68739d","Type":"ContainerDied","Data":"5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5"}
Nov 02 14:25:38.802230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.802199    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"
Nov 02 14:25:38.802230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.802229    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh" event={"ID":"ca5f7880-90a8-44cb-b8c1-191b5d68739d","Type":"ContainerDied","Data":"20c8281d76cd082d9ae16af48ba2257294ff0bc9f6661a7d406d6f2f1a94055b"}
Nov 02 14:25:38.802650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.802251    4492 scope.go:122] "RemoveContainer" containerID="5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5"
Nov 02 14:25:38.814505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.814480    4492 scope.go:122] "RemoveContainer" containerID="5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5"
Nov 02 14:25:38.814870 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:25:38.814850    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5\": container with ID starting with 5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5 not found: ID does not exist" containerID="5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5"
Nov 02 14:25:38.814939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.814877    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5"} err="failed to get container status \"5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5\": rpc error: code = NotFound desc = could not find container \"5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5\": container with ID starting with 5d345ecc9e0a62558fe6d1d5028e7c271fc5dff43625a7df1de71cc5db2e99b5 not found: ID does not exist"
Nov 02 14:25:38.849852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.849809    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"]
Nov 02 14:25:38.855571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:38.855544    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4dd54948-xjxbh"]
Nov 02 14:25:39.584133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:39.584097    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5f7880-90a8-44cb-b8c1-191b5d68739d" path="/var/lib/kubelet/pods/ca5f7880-90a8-44cb-b8c1-191b5d68739d/volumes"
Nov 02 14:25:40.102062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:40.102028    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:41.946389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:41.946322    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:42.356133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:42.356046    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:42.512382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:42.511623    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:43.821583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:43.821546    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:45.522682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:45.522645    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:46.517258 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:25:46.517211    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:25:47.272544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:47.272512    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:49.126513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:49.126472    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:49.315778 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:49.315750    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:49.953381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:49.951455    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:25:49.953381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:49.951533    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:25:49.973388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:49.970094    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:25:49.973388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:49.970171    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:25:50.032583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:50.032528    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:25:50.032907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:50.032857    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:25:51.456191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:51.456160    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:51.837670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:51.837575    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:51.955767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:51.955734    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:53.037646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.037616    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:53.148603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.148560    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:53.255167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.253797    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"]
Nov 02 14:25:53.255167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.254914    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ca5f7880-90a8-44cb-b8c1-191b5d68739d" containerName="payload-processing"
Nov 02 14:25:53.255167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.254940    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5f7880-90a8-44cb-b8c1-191b5d68739d" containerName="payload-processing"
Nov 02 14:25:53.255528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.255206    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5f7880-90a8-44cb-b8c1-191b5d68739d" containerName="payload-processing"
Nov 02 14:25:53.301393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.301273    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"]
Nov 02 14:25:53.301766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.301742    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.335615    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cfl-622143-provider-cred\""
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.335783    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\""
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.335870    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\""
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.335934    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-ca19ab63a2a2\""
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.336116    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\""
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.336181    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-gffbp\""
Nov 02 14:25:53.338846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.336349    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\""
Nov 02 14:25:53.386395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.386276    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.386969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.386348    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.386969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.386766    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.386969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.386846    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddnb\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.523166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.497602    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.529501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.529442    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddnb\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.529817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.529794    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.530083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.530055    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.530883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.524558    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.531921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.531728    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.571280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.571209    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddnb\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.597591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.589776    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.631077 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.631032    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:53.679385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.679335    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:53.828453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:53.828340    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:54.026023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:54.025978    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:54.110654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:54.109799    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"]
Nov 02 14:25:54.654411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:54.654371    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:55.015929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:55.015889    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt" event={"ID":"ff4ad729-3297-4813-b42e-6d0bfd5d5977","Type":"ContainerStarted","Data":"9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c"}
Nov 02 14:25:55.016152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:55.015963    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt" event={"ID":"ff4ad729-3297-4813-b42e-6d0bfd5d5977","Type":"ContainerStarted","Data":"0af19be79a3f952bef983cea0620ccf345aa298acd89036154439b01ecb0bf94"}
Nov 02 14:25:55.017049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:55.017027    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:25:55.061119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:55.061059    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt" podStartSLOduration=2.061039592 podStartE2EDuration="2.061039592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:55.060020178 +0000 UTC m=+1719.603735772" watchObservedRunningTime="2026-11-02 14:25:55.061039592 +0000 UTC m=+1719.604755187"
Nov 02 14:25:55.308771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:55.308674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:55.803212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:55.803178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:56.216623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:56.216592    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:56.335382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:56.335337    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:57.197153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:57.197119    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:57.358004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:57.357935    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:58.321670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:58.321637    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:58.939557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:58.939491    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:59.088604 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:59.088541    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:25:59.232288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:25:59.232190    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:00.767744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:00.767709    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:01.233216 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:01.233177    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:01.375992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:01.375959    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:02.646772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:02.646728    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:04.559305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:04.559271    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:06.440368 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:06.440330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:07.041741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:07.041704    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:26:08.134629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:08.134594    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:08.290002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:08.289970    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:09.822029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:09.821998    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:09.935849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:09.935814    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:10.860673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:10.860638    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:11.858750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:11.858721    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:13.196952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:13.196918    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:13.685343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:13.685310    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:15.454169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:15.454131    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:16.520945 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:26:16.520897    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:26:17.265532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:17.265501    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:17.476757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:17.476723    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:18.684132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:18.684097    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:18.883478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:18.883448    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:19.124632 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:19.124550    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:20.987174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:20.987132    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:21.130208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:21.130171    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:21.245456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:21.243784    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:22.012816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:22.012786    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:22.943535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:22.943498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:23.474980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:23.474945    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:24.605583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.605552    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"]
Nov 02 14:26:24.611450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.611421    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.630594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.630565    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"]
Nov 02 14:26:24.641394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.635283    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3261bdf2-serving-cert\""
Nov 02 14:26:24.682276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.682237    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.682705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.682642    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.683230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.683206    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfp9\" (UniqueName: \"kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.683438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.683418    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.699065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.699019    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:24.784805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.784778    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.785024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.784828    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.785085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.785022    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfp9\" (UniqueName: \"kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.785085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.785072    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.785231 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:26:24.785208    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-serving-cert: secret "maas-api-e2e-derive-3261bdf2-serving-cert" not found
Nov 02 14:26:24.785334 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:26:24.785317    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls podName:c53878e0-d97b-4abc-ae65-31f18f2a01de nodeName:}" failed. No retries permitted until 2026-11-02 14:26:25.285263545 +0000 UTC m=+1749.828979131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls") pod "maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" (UID: "c53878e0-d97b-4abc-ae65-31f18f2a01de") : secret "maas-api-e2e-derive-3261bdf2-serving-cert" not found
Nov 02 14:26:24.785601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.785580    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.830845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.824816    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.831746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.831714    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:24.861885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.861133    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfp9\" (UniqueName: \"kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:24.941585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:24.941550    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:25.292747 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:25.292711    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:25.295320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:25.295294    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\") pod \"maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:25.529156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:25.529121    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:25.979609 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:26:25.979568    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53878e0_d97b_4abc_ae65_31f18f2a01de.slice/crio-3d66c53d15268d9ad5a2b7e48eb2f8022a54a5b7b32fa73d76119a13c6a8a057 WatchSource:0}: Error finding container 3d66c53d15268d9ad5a2b7e48eb2f8022a54a5b7b32fa73d76119a13c6a8a057: Status 404 returned error can't find the container with id 3d66c53d15268d9ad5a2b7e48eb2f8022a54a5b7b32fa73d76119a13c6a8a057
Nov 02 14:26:25.987116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:25.986572    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"]
Nov 02 14:26:26.191686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:26.191607    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:26.308890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:26.308795    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:26.313816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:26.313771    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" event={"ID":"c53878e0-d97b-4abc-ae65-31f18f2a01de","Type":"ContainerStarted","Data":"3d66c53d15268d9ad5a2b7e48eb2f8022a54a5b7b32fa73d76119a13c6a8a057"}
Nov 02 14:26:26.808398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:26.808341    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:27.325413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:27.325349    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" event={"ID":"c53878e0-d97b-4abc-ae65-31f18f2a01de","Type":"ContainerStarted","Data":"b480666979b50c6fcba22738950f7c1380713f3209660cde8426343d5e0cc6b4"}
Nov 02 14:26:27.325838 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:27.325443    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:27.374550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:27.374500    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" podStartSLOduration=2.989873423 podStartE2EDuration="3.374484801s" totalImagesPullingTime="384.611378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:27.369527684 +0000 UTC m=+1751.913243278" watchObservedRunningTime="2026-11-02 14:26:27.374484801 +0000 UTC m=+1751.918200396"
Nov 02 14:26:27.393545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:27.393507    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:27.550226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:27.550193    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:28.958534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:28.958493    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:29.547210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:29.547176    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:30.790826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:30.790790    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:32.488627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:32.488594    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:33.342046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:33.341963    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:26:34.428630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:34.428598    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:36.229112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:36.229073    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:38.041850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:38.041817    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:39.921736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:39.921708    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:40.997579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:40.997541    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"]
Nov 02 14:26:41.868081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:41.868048    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:44.015468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:44.015435    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:45.997954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:45.997898    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt" podUID="ff4ad729-3297-4813-b42e-6d0bfd5d5977" containerName="payload-processing" containerID="cri-o://9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c" gracePeriod=25
Nov 02 14:26:46.021252 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.021225    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:46.308200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.308169    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:26:46.398097 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.398062    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb\") pod \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") "
Nov 02 14:26:46.398326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.398147    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume\") pod \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") "
Nov 02 14:26:46.398326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.398195    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume\") pod \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") "
Nov 02 14:26:46.398476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.398370    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials\") pod \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\" (UID: \"ff4ad729-3297-4813-b42e-6d0bfd5d5977\") "
Nov 02 14:26:46.398680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.398646    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume" pod "ff4ad729-3297-4813-b42e-6d0bfd5d5977" (UID: "ff4ad729-3297-4813-b42e-6d0bfd5d5977"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:46.398787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.398730    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume" pod "ff4ad729-3297-4813-b42e-6d0bfd5d5977" (UID: "ff4ad729-3297-4813-b42e-6d0bfd5d5977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:46.399178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.399152    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.399283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.399182    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4ad729-3297-4813-b42e-6d0bfd5d5977-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.400740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.400714    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb" pod "ff4ad729-3297-4813-b42e-6d0bfd5d5977" (UID: "ff4ad729-3297-4813-b42e-6d0bfd5d5977"). InnerVolumeSpecName "kube-api-access-cddnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:46.401327 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.401303    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials" pod "ff4ad729-3297-4813-b42e-6d0bfd5d5977" (UID: "ff4ad729-3297-4813-b42e-6d0bfd5d5977"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:46.497395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.497344    4492 generic.go:356] "Generic (PLEG): container finished" podID="ff4ad729-3297-4813-b42e-6d0bfd5d5977" containerID="9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c" exitCode=0
Nov 02 14:26:46.497662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.497441    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"
Nov 02 14:26:46.497777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.497458    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt" event={"ID":"ff4ad729-3297-4813-b42e-6d0bfd5d5977","Type":"ContainerDied","Data":"9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c"}
Nov 02 14:26:46.497984 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.497962    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt" event={"ID":"ff4ad729-3297-4813-b42e-6d0bfd5d5977","Type":"ContainerDied","Data":"0af19be79a3f952bef983cea0620ccf345aa298acd89036154439b01ecb0bf94"}
Nov 02 14:26:46.498057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.498008    4492 scope.go:122] "RemoveContainer" containerID="9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c"
Nov 02 14:26:46.500612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.500585    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddnb\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-kube-api-access-cddnb\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.500736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.500617    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff4ad729-3297-4813-b42e-6d0bfd5d5977-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.511040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.511002    4492 scope.go:122] "RemoveContainer" containerID="9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c"
Nov 02 14:26:46.511440 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:26:46.511399    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c\": container with ID starting with 9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c not found: ID does not exist" containerID="9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c"
Nov 02 14:26:46.511563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.511451    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c"} err="failed to get container status \"9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c\": rpc error: code = NotFound desc = could not find container \"9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c\": container with ID starting with 9834a0ae49398a26d1912afc16b74e557848f713333d0f1dcc86db4759e5105c not found: ID does not exist"
Nov 02 14:26:46.522886 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:26:46.522858    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:26:46.555977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.555887    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"]
Nov 02 14:26:46.561165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:46.561128    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b8964x4pt"]
Nov 02 14:26:47.584943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:47.584897    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4ad729-3297-4813-b42e-6d0bfd5d5977" path="/var/lib/kubelet/pods/ff4ad729-3297-4813-b42e-6d0bfd5d5977/volumes"
Nov 02 14:26:47.926182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:47.926146    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:49.747045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:49.746999    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:51.882639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:51.882593    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:53.187336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:53.187302    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:54.062064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:54.062026    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:55.467569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:55.467533    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:56.307289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:56.307238    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:26:58.103158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:26:58.103126    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:00.093207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:00.093170    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:02.000148 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:02.000109    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:03.684997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:03.684963    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:05.474929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:05.474884    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:07.331773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:07.331738    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:08.391911 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:08.391868    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:09.216899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:09.216851    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:10.243646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:10.243611    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:10.374398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:10.373872    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:11.130187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:11.130157    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:11.759732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:11.759694    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:12.945337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:12.945300    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:13.278414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:13.278317    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:14.741801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:14.741764    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:16.274096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:16.274060    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:16.517680 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:27:16.517638    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:27:16.670410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:16.670307    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:16.787289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:16.787258    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:18.427574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:18.427539    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:18.666688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:18.666662    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:20.134718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:20.134681    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:21.700409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:21.699687    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"]
Nov 02 14:27:21.700409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:21.699945    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" podUID="c53878e0-d97b-4abc-ae65-31f18f2a01de" containerName="maas-api" containerID="cri-o://b480666979b50c6fcba22738950f7c1380713f3209660cde8426343d5e0cc6b4" gracePeriod=30
Nov 02 14:27:21.749822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:21.749675    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:21.866586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:21.866550    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:21.998499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:21.989483    4492 generic.go:356] "Generic (PLEG): container finished" podID="c53878e0-d97b-4abc-ae65-31f18f2a01de" containerID="b480666979b50c6fcba22738950f7c1380713f3209660cde8426343d5e0cc6b4" exitCode=0
Nov 02 14:27:22.007555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.007516    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" event={"ID":"c53878e0-d97b-4abc-ae65-31f18f2a01de","Type":"ContainerDied","Data":"b480666979b50c6fcba22738950f7c1380713f3209660cde8426343d5e0cc6b4"}
Nov 02 14:27:22.255419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.219599    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:27:22.255419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.238783    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9\" (UniqueName: \"kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9\") pod \"c53878e0-d97b-4abc-ae65-31f18f2a01de\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") "
Nov 02 14:27:22.255419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.238879    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs\") pod \"c53878e0-d97b-4abc-ae65-31f18f2a01de\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") "
Nov 02 14:27:22.293391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.287327    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9" pod "c53878e0-d97b-4abc-ae65-31f18f2a01de" (UID: "c53878e0-d97b-4abc-ae65-31f18f2a01de"). InnerVolumeSpecName "kube-api-access-4jfp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:22.310055 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.310011    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs" pod "c53878e0-d97b-4abc-ae65-31f18f2a01de" (UID: "c53878e0-d97b-4abc-ae65-31f18f2a01de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:27:22.319615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.319582    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:22.356614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.355790    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca\") pod \"c53878e0-d97b-4abc-ae65-31f18f2a01de\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") "
Nov 02 14:27:22.356614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.355875    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\") pod \"c53878e0-d97b-4abc-ae65-31f18f2a01de\" (UID: \"c53878e0-d97b-4abc-ae65-31f18f2a01de\") "
Nov 02 14:27:22.356614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.356269    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca" pod "c53878e0-d97b-4abc-ae65-31f18f2a01de" (UID: "c53878e0-d97b-4abc-ae65-31f18f2a01de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:22.356938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.356751    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jfp9\" (UniqueName: \"kubernetes.io/projected/c53878e0-d97b-4abc-ae65-31f18f2a01de-kube-api-access-4jfp9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:22.356938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.356776    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:22.356938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.356788    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53878e0-d97b-4abc-ae65-31f18f2a01de-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:22.359164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.359139    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls" pod "c53878e0-d97b-4abc-ae65-31f18f2a01de" (UID: "c53878e0-d97b-4abc-ae65-31f18f2a01de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:27:22.439787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.439731    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:22.458494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.458462    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c53878e0-d97b-4abc-ae65-31f18f2a01de-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:22.567535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.566950    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:22.696934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:22.696901    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:23.007944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.007911    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"
Nov 02 14:27:23.008407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.007952    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw" event={"ID":"c53878e0-d97b-4abc-ae65-31f18f2a01de","Type":"ContainerDied","Data":"3d66c53d15268d9ad5a2b7e48eb2f8022a54a5b7b32fa73d76119a13c6a8a057"}
Nov 02 14:27:23.008407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.008022    4492 scope.go:122] "RemoveContainer" containerID="b480666979b50c6fcba22738950f7c1380713f3209660cde8426343d5e0cc6b4"
Nov 02 14:27:23.061742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.061702    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"]
Nov 02 14:27:23.068568 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.068534    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3261bdf2-58654cccdc-vksgw"]
Nov 02 14:27:23.585027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.584995    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53878e0-d97b-4abc-ae65-31f18f2a01de" path="/var/lib/kubelet/pods/c53878e0-d97b-4abc-ae65-31f18f2a01de/volumes"
Nov 02 14:27:23.606998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:23.606971    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:24.067215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:24.067164    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:24.572606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:24.572069    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:24.727957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:24.727921    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:26.343000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:26.342967    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:27.817153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:27.817114    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:27.943491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:27.943451    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:28.315126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:28.315090    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:29.536348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.536301    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:29.709506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.709446    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:29.718950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.718920    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"]
Nov 02 14:27:29.719764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.719748    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ff4ad729-3297-4813-b42e-6d0bfd5d5977" containerName="payload-processing"
Nov 02 14:27:29.719832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.719765    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4ad729-3297-4813-b42e-6d0bfd5d5977" containerName="payload-processing"
Nov 02 14:27:29.719832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.719803    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c53878e0-d97b-4abc-ae65-31f18f2a01de" containerName="maas-api"
Nov 02 14:27:29.719832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.719808    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53878e0-d97b-4abc-ae65-31f18f2a01de" containerName="maas-api"
Nov 02 14:27:29.719928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.719912    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4ad729-3297-4813-b42e-6d0bfd5d5977" containerName="payload-processing"
Nov 02 14:27:29.719928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.719924    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53878e0-d97b-4abc-ae65-31f18f2a01de" containerName="maas-api"
Nov 02 14:27:29.724893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.724877    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:29.745176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.745124    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"]
Nov 02 14:27:29.746499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.746475    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f72bc6\""
Nov 02 14:27:29.747511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.747490    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f72bc6-dockercfg-zp7xz\""
Nov 02 14:27:29.826520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.825844    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:29.826520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.826131    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78mn\" (UniqueName: \"kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn\") pod \"payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:29.986382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.960516    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:29.986382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.960830    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78mn\" (UniqueName: \"kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn\") pod \"payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:29.986382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.962262    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:29.999489 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:29.999416    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78mn\" (UniqueName: \"kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn\") pod \"payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:30.048348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:30.048314    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:30.350414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:30.350378    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:30.363971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:30.363946    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"]
Nov 02 14:27:30.366339 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:27:30.366306    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87eabb3c_979d_43c2_abb6_dc305fe31238.slice/crio-4acb24c3a3cecca540acf02df478480f10daa910a3b1332d8adf1c7f411081b6 WatchSource:0}: Error finding container 4acb24c3a3cecca540acf02df478480f10daa910a3b1332d8adf1c7f411081b6: Status 404 returned error can't find the container with id 4acb24c3a3cecca540acf02df478480f10daa910a3b1332d8adf1c7f411081b6
Nov 02 14:27:31.325025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:31.324989    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445" event={"ID":"87eabb3c-979d-43c2-abb6-dc305fe31238","Type":"ContainerStarted","Data":"b9a454f0d2d775023867e1e0eb5929b3d76cb24c7966610246a7a946f62278fc"}
Nov 02 14:27:31.325459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:31.325045    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445" event={"ID":"87eabb3c-979d-43c2-abb6-dc305fe31238","Type":"ContainerStarted","Data":"4acb24c3a3cecca540acf02df478480f10daa910a3b1332d8adf1c7f411081b6"}
Nov 02 14:27:31.325459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:31.325190    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:31.364826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:31.364763    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445" podStartSLOduration=2.364750524 podStartE2EDuration="2.364750524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:31.361099938 +0000 UTC m=+1815.904815537" watchObservedRunningTime="2026-11-02 14:27:31.364750524 +0000 UTC m=+1815.908466116"
Nov 02 14:27:31.969786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:31.969750    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:32.082078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:32.082044    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:32.208093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:32.208045    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:33.959798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:33.959752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:35.843113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:35.843059    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:37.640625 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:37.640589    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:38.885126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:38.885084    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:39.025451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:39.025407    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:39.570447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:39.570411    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:40.546963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:40.546921    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:40.713492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:40.713453    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"]
Nov 02 14:27:40.713716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:40.713694    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" containerID="cri-o://65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911" gracePeriod=30
Nov 02 14:27:40.927828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:40.927794    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"]
Nov 02 14:27:40.928086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:40.928046    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy" containerID="cri-o://67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6" gracePeriod=30
Nov 02 14:27:41.526624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:41.526594    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:42.336048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:42.336018    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:27:43.414560 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:43.414524    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:45.021653 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:45.021612    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:45.173995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:45.173952    4492 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:27:45.174175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:45.174026    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:27:45.867002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:45.866967    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:46.071787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.071716    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:27:46.193526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193488    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.193744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193584    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.193744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193636    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.193744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193674    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.193954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193798    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.193954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193940    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:46.194070 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.193958    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.194070 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.194001    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.194180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.194119    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.194180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.194162    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token\") pod \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\" (UID: \"c0f05d9b-6d57-43fc-a8f1-261f676d8678\") "
Nov 02 14:27:46.194289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.194275    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.194533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.194466    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.194873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.194852    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.195085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.195062    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.195166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.195092    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.195166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.195108    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.195633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.195612    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.200510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.200474    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "kube-api-access-666tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:46.200663 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.200643    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.202439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.202403    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:46.202439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.202419    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo" pod "c0f05d9b-6d57-43fc-a8f1-261f676d8678" (UID: "c0f05d9b-6d57-43fc-a8f1-261f676d8678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:27:46.237395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.237308    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:27:46.296789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.296711    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-666tf\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-kube-api-access-666tf\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.296789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.296737    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.296789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.296748    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.296789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.296756    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.296789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.296765    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.296789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.296775    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f05d9b-6d57-43fc-a8f1-261f676d8678-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.397389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397337    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397503    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397527    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397596    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397631    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397663    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397737    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397855    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.397955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397884    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.398018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.397985    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data\") pod \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\" (UID: \"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a\") "
Nov 02 14:27:46.398253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398090    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.398581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398505    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.398794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398767    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.398912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398858    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.398912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398877    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.398912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398886    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.398912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398894    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.399220 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.398936    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:46.410609 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.410569    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:27:46.410609 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.410592    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:27:46.410830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.410629    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:46.410830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.410659    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8" pod "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" (UID: "2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a"). InnerVolumeSpecName "kube-api-access-4kdz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:46.499849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.499808    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.499849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.499835    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.499849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.499848    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdz8\" (UniqueName: \"kubernetes.io/projected/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-kube-api-access-4kdz8\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.500131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.499862    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.500131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.499871    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.518768 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:27:46.518729    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:27:46.592268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.592177    4492 generic.go:356] "Generic (PLEG): container finished" podID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerID="67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6" exitCode=0
Nov 02 14:27:46.592268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.592225    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"
Nov 02 14:27:46.592718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.592260    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" event={"ID":"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a","Type":"ContainerDied","Data":"67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6"}
Nov 02 14:27:46.592718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.592330    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc" event={"ID":"2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a","Type":"ContainerDied","Data":"7665c0bad7c58532fda9fb7b858aad82316379d3d107fde7c5c0ffc231caa893"}
Nov 02 14:27:46.592718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.592400    4492 scope.go:122] "RemoveContainer" containerID="67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6"
Nov 02 14:27:46.594067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.594045    4492 generic.go:356] "Generic (PLEG): container finished" podID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerID="65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911" exitCode=0
Nov 02 14:27:46.594180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.594075    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" event={"ID":"c0f05d9b-6d57-43fc-a8f1-261f676d8678","Type":"ContainerDied","Data":"65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911"}
Nov 02 14:27:46.594180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.594098    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"
Nov 02 14:27:46.594180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.594122    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t" event={"ID":"c0f05d9b-6d57-43fc-a8f1-261f676d8678","Type":"ContainerDied","Data":"909529b7b7d489d6d275b1efce03656d25ba5a25013fa61c538fb3d33a36e95e"}
Nov 02 14:27:46.610428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.610405    4492 scope.go:122] "RemoveContainer" containerID="67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6"
Nov 02 14:27:46.610957 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:27:46.610932    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6\": container with ID starting with 67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6 not found: ID does not exist" containerID="67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6"
Nov 02 14:27:46.611080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.610962    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6"} err="failed to get container status \"67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6\": rpc error: code = NotFound desc = could not find container \"67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6\": container with ID starting with 67d75e119ee3ed23cd452274234ea50ddb0244e652037fb7a54f7206ea6780d6 not found: ID does not exist"
Nov 02 14:27:46.611080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.610982    4492 scope.go:122] "RemoveContainer" containerID="65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911"
Nov 02 14:27:46.621159 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.621136    4492 scope.go:122] "RemoveContainer" containerID="65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911"
Nov 02 14:27:46.621549 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:27:46.621524    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911\": container with ID starting with 65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911 not found: ID does not exist" containerID="65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911"
Nov 02 14:27:46.621626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.621560    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911"} err="failed to get container status \"65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911\": rpc error: code = NotFound desc = could not find container \"65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911\": container with ID starting with 65245ec28d09d85419ed7d2778fe99bf8044d76397b4015b6c451c6d7b1c3911 not found: ID does not exist"
Nov 02 14:27:46.644286 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.644250    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"]
Nov 02 14:27:46.654310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.654278    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-znwrc"]
Nov 02 14:27:46.680319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.680288    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"]
Nov 02 14:27:46.685137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:46.685109    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-b5t2t"]
Nov 02 14:27:47.582260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:47.582225    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" path="/var/lib/kubelet/pods/2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a/volumes"
Nov 02 14:27:47.582834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:47.582809    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" path="/var/lib/kubelet/pods/c0f05d9b-6d57-43fc-a8f1-261f676d8678/volumes"
Nov 02 14:27:47.864050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:47.863897    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:47.980217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:47.980179    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:48.157205 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:48.157112    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:48.876565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:48.876522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:49.282233 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:49.282200    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:49.926865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:49.926824    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:50.067015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:50.066979    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:50.436913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:50.436878    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:50.835536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:50.835453    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:51.824005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:51.823972    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:53.594531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:53.594497    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:53.761558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:53.759920    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:53.873111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:53.873023    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:54.805399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:54.790885    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:54.911759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:54.911724    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:55.857219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:55.857189    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:56.645906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:56.645869    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:56.759122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:56.759086    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:56.886811 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:56.886780    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:57.504395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.494722    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:57.787806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.787729    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"]
Nov 02 14:27:57.788434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.788416    4492 cpu_manager.go:414] "containerMap: removing container" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy"
Nov 02 14:27:57.788434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.788432    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy"
Nov 02 14:27:57.788606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.788448    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy"
Nov 02 14:27:57.788606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.788454    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy"
Nov 02 14:27:57.788606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.788553    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee95c24-51dc-4764-b4bf-7e1eae8a0e4a" containerName="istio-proxy"
Nov 02 14:27:57.788606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.788566    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f05d9b-6d57-43fc-a8f1-261f676d8678" containerName="istio-proxy"
Nov 02 14:27:57.793780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.793759    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:57.818794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.818565    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"routing-overlay\""
Nov 02 14:27:57.818794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.818665    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"openshift-service-ca.crt\""
Nov 02 14:27:57.818794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.818757    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ba664\""
Nov 02 14:27:57.826979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.826939    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"]
Nov 02 14:27:57.896384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.891799    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-e2e--d8f5123f-dockercfg-qlpzh\""
Nov 02 14:27:57.896384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.891934    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"kube-root-ca.crt\""
Nov 02 14:27:57.896384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.892047    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-d6506fd29d97\""
Nov 02 14:27:57.896384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.892132    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-b8b0c81894ae\""
Nov 02 14:27:57.896384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.892186    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-598f75ee016d\""
Nov 02 14:27:57.930805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.930765    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6k6p\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:57.952300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.931052    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:57.955982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.955436    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:57.955982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:57.955530    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.057749 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.057659    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:58.058076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.058041    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6k6p\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.058193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.058097    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.058193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.058142    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.058193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.058168    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.059095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.059068    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.061059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.060260    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.064371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.064329    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.081168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.081130    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6k6p\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.113695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.113654    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:58.501554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.500922    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"]
Nov 02 14:27:58.686143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.686110    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"]
Nov 02 14:27:58.701182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.692061    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.708948 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.708921    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"]
Nov 02 14:27:58.714339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.714301    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"openshift-service-ca.crt\""
Nov 02 14:27:58.714511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.714339    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12b8da\""
Nov 02 14:27:58.714511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.714445    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-f36927adbb8c\""
Nov 02 14:27:58.714511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.714465    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-dec30ec8ff5a\""
Nov 02 14:27:58.714674 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.714525    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-047606d198bb\""
Nov 02 14:27:58.715579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.715557    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-e2e--3ffe20a0-dockercfg-kwvdn\""
Nov 02 14:27:58.715712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.715607    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"routing-overlay\""
Nov 02 14:27:58.715712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.715661    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"kube-root-ca.crt\""
Nov 02 14:27:58.798537 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.798457    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.798537 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.798513    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6wd\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.798747 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.798635    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.798924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.798905    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.900209 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.900166    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.900707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.900309    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.900707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.900376    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6wd\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.900707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.900465    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.901086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.901056    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.901239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.901152    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.903247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.903228    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:58.922063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:58.922036    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6wd\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:59.012901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.012857    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:27:59.032322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.032293    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" event={"ID":"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4","Type":"ContainerStarted","Data":"bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83"}
Nov 02 14:27:59.032479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.032348    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" event={"ID":"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4","Type":"ContainerStarted","Data":"ed7272ad5c8a98df4b95502c10c19906157e855635ec0fae94433613297c7356"}
Nov 02 14:27:59.032479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.032408    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:27:59.074110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.073769    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" podStartSLOduration=2.073746836 podStartE2EDuration="2.073746836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:59.07085173 +0000 UTC m=+1843.614567324" watchObservedRunningTime="2026-11-02 14:27:59.073746836 +0000 UTC m=+1843.617462430"
Nov 02 14:27:59.350867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.350759    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"]
Nov 02 14:27:59.353840 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:27:59.353811    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b0402b_d291_46e6_977f_7fe1aa2f60b3.slice/crio-94623246084ca1c21a0b0a9927675c4800025fc75bafacb393c937ef44ad3811 WatchSource:0}: Error finding container 94623246084ca1c21a0b0a9927675c4800025fc75bafacb393c937ef44ad3811: Status 404 returned error can't find the container with id 94623246084ca1c21a0b0a9927675c4800025fc75bafacb393c937ef44ad3811
Nov 02 14:27:59.762024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.761991    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:27:59.901379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:27:59.899468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:00.096695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:00.096170    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:00.167430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:00.158689    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" event={"ID":"f7b0402b-d291-46e6-977f-7fe1aa2f60b3","Type":"ContainerStarted","Data":"27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00"}
Nov 02 14:28:00.167430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:00.158773    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" event={"ID":"f7b0402b-d291-46e6-977f-7fe1aa2f60b3","Type":"ContainerStarted","Data":"94623246084ca1c21a0b0a9927675c4800025fc75bafacb393c937ef44ad3811"}
Nov 02 14:28:00.167430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:00.165417    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:28:00.198245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:00.198182    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" podStartSLOduration=2.198164277 podStartE2EDuration="2.198164277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:00.194752798 +0000 UTC m=+1844.738468391" watchObservedRunningTime="2026-11-02 14:28:00.198164277 +0000 UTC m=+1844.741879874"
Nov 02 14:28:00.219541 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.219501    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-12b8da/agc-cred-f36927adbb8c: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:00.219541 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.219539    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:00.219782 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.219604    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials podName:f7b0402b-d291-46e6-977f-7fe1aa2f60b3 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:00.719583 +0000 UTC m=+1845.263298585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3") : secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:00.219860 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.219817    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ba664/agc-cred-d6506fd29d97: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:00.219860 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.219854    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:00.219969 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.219902    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials podName:2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:00.719885395 +0000 UTC m=+1845.263600982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4") : secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:00.726391 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.726332    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-12b8da/agc-cred-f36927adbb8c: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:00.726391 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.726375    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ba664/agc-cred-d6506fd29d97: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:00.726391 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.726401    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:00.726675 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.726384    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:00.726675 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.726467    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials podName:2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:01.726452275 +0000 UTC m=+1846.270167860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4") : secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:00.726675 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:00.726522    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials podName:f7b0402b-d291-46e6-977f-7fe1aa2f60b3 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:01.726502652 +0000 UTC m=+1846.270218238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3") : secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:01.229450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:01.229409    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"]
Nov 02 14:28:01.750729 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:01.750691    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ba664/agc-cred-d6506fd29d97: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:01.750729 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:01.750723    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:01.750994 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:01.750791    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-12b8da/agc-cred-f36927adbb8c: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:01.750994 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:01.750811    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:01.750994 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:01.750802    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials podName:2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:03.75078521 +0000 UTC m=+1848.294500799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4") : secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:01.750994 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:01.750855    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials podName:f7b0402b-d291-46e6-977f-7fe1aa2f60b3 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:03.750841145 +0000 UTC m=+1848.294556730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3") : secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:01.977551 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:01.977515    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:02.135426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:02.135299    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"]
Nov 02 14:28:03.708117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:03.708074    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:03.786015 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:03.785975    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-12b8da/agc-cred-f36927adbb8c: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:03.786272 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:03.786251    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:03.786472 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:03.786459    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials podName:f7b0402b-d291-46e6-977f-7fe1aa2f60b3 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:07.786436177 +0000 UTC m=+1852.330151763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3") : secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:03.787050 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:03.787015    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9ba664/agc-cred-d6506fd29d97: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:03.787182 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:03.787172    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6: secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:03.787305 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:03.787296    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials podName:2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:07.787276954 +0000 UTC m=+1852.330992539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4") : secret "agc-cred-d6506fd29d97" not found
Nov 02 14:28:05.544944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:05.544903    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:06.230117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.230069    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" podUID="2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" containerName="payload-processing" containerID="cri-o://bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83" gracePeriod=25
Nov 02 14:28:06.521846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.521822    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:28:06.629825 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.629794    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p\") pod \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") "
Nov 02 14:28:06.630263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630078    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume\") pod \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") "
Nov 02 14:28:06.630263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630219    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials\") pod \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") "
Nov 02 14:28:06.630416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630272    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume\") pod \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\" (UID: \"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4\") "
Nov 02 14:28:06.630495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630468    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume" pod "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:06.630728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630706    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume" pod "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:06.631067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630906    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:06.631067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.630918    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:06.632207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.632183    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p" pod "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4"). InnerVolumeSpecName "kube-api-access-h6k6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:06.632495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.632463    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials" pod "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" (UID: "2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:06.732719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.732674    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:06.732719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:06.732716    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6k6p\" (UniqueName: \"kubernetes.io/projected/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4-kube-api-access-h6k6p\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:07.201313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.201277    4492 generic.go:356] "Generic (PLEG): container finished" podID="2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" containerID="bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83" exitCode=0
Nov 02 14:28:07.201528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.201335    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" event={"ID":"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4","Type":"ContainerDied","Data":"bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83"}
Nov 02 14:28:07.201528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.201340    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"
Nov 02 14:28:07.201528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.201403    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6" event={"ID":"2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4","Type":"ContainerDied","Data":"ed7272ad5c8a98df4b95502c10c19906157e855635ec0fae94433613297c7356"}
Nov 02 14:28:07.201528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.201424    4492 scope.go:122] "RemoveContainer" containerID="bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83"
Nov 02 14:28:07.212617 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.212596    4492 scope.go:122] "RemoveContainer" containerID="bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83"
Nov 02 14:28:07.212920 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:07.212899    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83\": container with ID starting with bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83 not found: ID does not exist" containerID="bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83"
Nov 02 14:28:07.212962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.212930    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83"} err="failed to get container status \"bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83\": rpc error: code = NotFound desc = could not find container \"bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83\": container with ID starting with bbbf6a4126398a04a85070cc19a9a5b5731b47678bf435cc1cc3777350724a83 not found: ID does not exist"
Nov 02 14:28:07.259324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.259285    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"]
Nov 02 14:28:07.268454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.268423    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6d5ddw7j6"]
Nov 02 14:28:07.336532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.336501    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:07.586044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:07.585958    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" path="/var/lib/kubelet/pods/2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4/volumes"
Nov 02 14:28:07.849816 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:07.849717    4492 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-12b8da/agc-cred-f36927adbb8c: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:07.849816 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:07.849747    4492 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h: secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:07.850459 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:07.849846    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials podName:f7b0402b-d291-46e6-977f-7fe1aa2f60b3 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:15.849805072 +0000 UTC m=+1860.393520661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3") : secret "agc-cred-f36927adbb8c" not found
Nov 02 14:28:08.168699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.168597    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" podUID="f7b0402b-d291-46e6-977f-7fe1aa2f60b3" containerName="payload-processing" containerID="cri-o://27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00" gracePeriod=25
Nov 02 14:28:08.445546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.445516    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:28:08.513760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.513727    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:08.559427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.559393    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume\") pod \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") "
Nov 02 14:28:08.559664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.559478    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume\") pod \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") "
Nov 02 14:28:08.559664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.559597    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd\") pod \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") "
Nov 02 14:28:08.559792 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.559686    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials\") pod \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\" (UID: \"f7b0402b-d291-46e6-977f-7fe1aa2f60b3\") "
Nov 02 14:28:08.559871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.559840    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume" pod "f7b0402b-d291-46e6-977f-7fe1aa2f60b3" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:08.559975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.559950    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume" pod "f7b0402b-d291-46e6-977f-7fe1aa2f60b3" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:08.560504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.560412    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:08.560771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.560734    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:08.562295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.562271    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials" pod "f7b0402b-d291-46e6-977f-7fe1aa2f60b3" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:08.562442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.562331    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd" pod "f7b0402b-d291-46e6-977f-7fe1aa2f60b3" (UID: "f7b0402b-d291-46e6-977f-7fe1aa2f60b3"). InnerVolumeSpecName "kube-api-access-kx6wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:08.662076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.662039    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6wd\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-kube-api-access-kx6wd\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:08.662076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:08.662066    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b0402b-d291-46e6-977f-7fe1aa2f60b3-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:09.020961 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.020926    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:09.220186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.220149    4492 generic.go:356] "Generic (PLEG): container finished" podID="f7b0402b-d291-46e6-977f-7fe1aa2f60b3" containerID="27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00" exitCode=0
Nov 02 14:28:09.220400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.220205    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"
Nov 02 14:28:09.220400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.220236    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" event={"ID":"f7b0402b-d291-46e6-977f-7fe1aa2f60b3","Type":"ContainerDied","Data":"27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00"}
Nov 02 14:28:09.220400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.220294    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h" event={"ID":"f7b0402b-d291-46e6-977f-7fe1aa2f60b3","Type":"ContainerDied","Data":"94623246084ca1c21a0b0a9927675c4800025fc75bafacb393c937ef44ad3811"}
Nov 02 14:28:09.220400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.220315    4492 scope.go:122] "RemoveContainer" containerID="27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00"
Nov 02 14:28:09.232063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.232038    4492 scope.go:122] "RemoveContainer" containerID="27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00"
Nov 02 14:28:09.232343 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:09.232319    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00\": container with ID starting with 27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00 not found: ID does not exist" containerID="27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00"
Nov 02 14:28:09.232446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.232375    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00"} err="failed to get container status \"27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00\": rpc error: code = NotFound desc = could not find container \"27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00\": container with ID starting with 27184e7892695c38dfb6fadcd2aaa2c363d4db85387ece32be83f19606b0cc00 not found: ID does not exist"
Nov 02 14:28:09.270823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.270790    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"]
Nov 02 14:28:09.278633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.278547    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-688f6h52h"]
Nov 02 14:28:09.582934 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:09.582848    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b0402b-d291-46e6-977f-7fe1aa2f60b3" path="/var/lib/kubelet/pods/f7b0402b-d291-46e6-977f-7fe1aa2f60b3/volumes"
Nov 02 14:28:10.994735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:10.994697    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:12.966489 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:12.966454    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:13.440815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:13.440784    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:14.862502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:14.862447    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:16.518757 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:16.518717    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:28:16.662144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:16.662104    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:18.727259 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:18.727225    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:20.742512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:20.742477    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:20.856777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:20.856731    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:22.750491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:22.750457    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:23.184500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:23.184470    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:23.325059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:23.325026    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:23.730282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:23.730251    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:24.864892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:24.864860    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:26.854956 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:26.854915    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:28.602378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:28.602327    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:28.606434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:28.606405    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"]
Nov 02 14:28:28.606633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:28.606614    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy" containerID="cri-o://b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273" gracePeriod=30
Nov 02 14:28:29.144555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:29.144519    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:29.793610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:29.793558    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:30.000508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.000476    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:30.127000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.126906    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:30.527244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.527214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:30.661378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.657695    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:30.958686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.958633    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"]
Nov 02 14:28:30.960346 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.960315    4492 cpu_manager.go:414] "containerMap: removing container" podUID="2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" containerName="payload-processing"
Nov 02 14:28:30.960536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.960522    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" containerName="payload-processing"
Nov 02 14:28:30.960656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.960645    4492 cpu_manager.go:414] "containerMap: removing container" podUID="f7b0402b-d291-46e6-977f-7fe1aa2f60b3" containerName="payload-processing"
Nov 02 14:28:30.960734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.960724    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b0402b-d291-46e6-977f-7fe1aa2f60b3" containerName="payload-processing"
Nov 02 14:28:30.961017 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.961003    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1135dd-f3ed-49fd-a7fd-4bf944f65ad4" containerName="payload-processing"
Nov 02 14:28:30.961123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.961112    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b0402b-d291-46e6-977f-7fe1aa2f60b3" containerName="payload-processing"
Nov 02 14:28:30.980478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:30.978634    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.000928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.000897    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"]
Nov 02 14:28:31.026765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.026732    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\""
Nov 02 14:28:31.027074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027052    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-gffbp\""
Nov 02 14:28:31.027278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027260    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cfl-622143-provider-cred\""
Nov 02 14:28:31.027544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027528    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\""
Nov 02 14:28:31.027660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027163    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-ca19ab63a2a2\""
Nov 02 14:28:31.032414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027683    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\""
Nov 02 14:28:31.032414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027622    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-0a71f3a9867d\""
Nov 02 14:28:31.032414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.027805    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-8434b5bb1061\""
Nov 02 14:28:31.049701 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.049665    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\""
Nov 02 14:28:31.128997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.128954    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.130173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.130125    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.130619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.130556    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ln7\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.130619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.130634    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.181437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.181395    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:31.231708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.231630    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ln7\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.231708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.231675    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.231708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.231706    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.231935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.231753    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.232330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.232310    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.232573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.232549    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.234818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.234794    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.268544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.268507    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ln7\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.439378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.427625    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:31.836565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:31.836534    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"]
Nov 02 14:28:31.838213 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:28:31.838174    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b61b82_9366_41ff_98e2_6930a32829d4.slice/crio-2cba679900508f2ba1e58ededdc6ae6b093c1b5f318cc876c373d1d2107d8b92 WatchSource:0}: Error finding container 2cba679900508f2ba1e58ededdc6ae6b093c1b5f318cc876c373d1d2107d8b92: Status 404 returned error can't find the container with id 2cba679900508f2ba1e58ededdc6ae6b093c1b5f318cc876c373d1d2107d8b92
Nov 02 14:28:32.521594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:32.521560    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q" event={"ID":"55b61b82-9366-41ff-98e2-6930a32829d4","Type":"ContainerStarted","Data":"d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b"}
Nov 02 14:28:32.522036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:32.521612    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q" event={"ID":"55b61b82-9366-41ff-98e2-6930a32829d4","Type":"ContainerStarted","Data":"2cba679900508f2ba1e58ededdc6ae6b093c1b5f318cc876c373d1d2107d8b92"}
Nov 02 14:28:32.522036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:32.521772    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:32.560732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:32.560671    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q" podStartSLOduration=2.560655513 podStartE2EDuration="2.560655513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:32.560044504 +0000 UTC m=+1877.103760098" watchObservedRunningTime="2026-11-02 14:28:32.560655513 +0000 UTC m=+1877.104371105"
Nov 02 14:28:32.713088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:32.713051    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:33.879868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:33.879809    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:28:34.005102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005065    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005190    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005233    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005286    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005375    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005396    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:28:34.005498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005470    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005522    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:28:34.005650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005554    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005595    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005707    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b\") pod \"0da14694-3a1f-4327-aa17-77ec3cd9549f\" (UID: \"0da14694-3a1f-4327-aa17-77ec3cd9549f\") "
Nov 02 14:28:34.005795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005712    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:28:34.005896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.005809    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:28:34.006194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.006169    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:34.006620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.006340    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.006620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.006432    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.006620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.006445    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.006620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.006468    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da14694-3a1f-4327-aa17-77ec3cd9549f-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.006620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.006579    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.008119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.008094    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:34.008221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.008175    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:28:34.008420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.008400    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:28:34.008594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.008575    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b" pod "0da14694-3a1f-4327-aa17-77ec3cd9549f" (UID: "0da14694-3a1f-4327-aa17-77ec3cd9549f"). InnerVolumeSpecName "kube-api-access-z7k9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:34.108289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.108258    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.108289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.108285    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.108544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.108299    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da14694-3a1f-4327-aa17-77ec3cd9549f-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.108544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.108312    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7k9b\" (UniqueName: \"kubernetes.io/projected/0da14694-3a1f-4327-aa17-77ec3cd9549f-kube-api-access-z7k9b\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:28:34.543137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.543095    4492 generic.go:356] "Generic (PLEG): container finished" podID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerID="b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273" exitCode=0
Nov 02 14:28:34.543367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.543148    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"
Nov 02 14:28:34.543367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.543186    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" event={"ID":"0da14694-3a1f-4327-aa17-77ec3cd9549f","Type":"ContainerDied","Data":"b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273"}
Nov 02 14:28:34.543367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.543253    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7" event={"ID":"0da14694-3a1f-4327-aa17-77ec3cd9549f","Type":"ContainerDied","Data":"1e01de09952d8c4721a81e71be304659476931d662b6bb5df53e96a575fcaacf"}
Nov 02 14:28:34.543367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.543282    4492 scope.go:122] "RemoveContainer" containerID="b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273"
Nov 02 14:28:34.556557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.556539    4492 scope.go:122] "RemoveContainer" containerID="f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"
Nov 02 14:28:34.568876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.567295    4492 scope.go:122] "RemoveContainer" containerID="b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273"
Nov 02 14:28:34.571509 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:34.571477    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273\": container with ID starting with b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273 not found: ID does not exist" containerID="b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273"
Nov 02 14:28:34.571650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.571522    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273"} err="failed to get container status \"b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273\": rpc error: code = NotFound desc = could not find container \"b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273\": container with ID starting with b27a6fae8297415759924efdd141eae9dc8ad4f3471f0c17710faabdcb467273 not found: ID does not exist"
Nov 02 14:28:34.571650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.571547    4492 scope.go:122] "RemoveContainer" containerID="f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"
Nov 02 14:28:34.571855 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:34.571835    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd\": container with ID starting with f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd not found: ID does not exist" containerID="f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"
Nov 02 14:28:34.571918 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.571873    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd"} err="failed to get container status \"f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd\": rpc error: code = NotFound desc = could not find container \"f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd\": container with ID starting with f24b4b74739ac7584ae9dacddcb519d8ac574773244a2f84ade259db2fb53fdd not found: ID does not exist"
Nov 02 14:28:34.603071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.603032    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"]
Nov 02 14:28:34.612877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.612830    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-6z2t7"]
Nov 02 14:28:34.726872 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:34.726835    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:35.585230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:35.585200    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" path="/var/lib/kubelet/pods/0da14694-3a1f-4327-aa17-77ec3cd9549f/volumes"
Nov 02 14:28:36.578175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:36.578139    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:38.320425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:38.320389    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:40.112790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:40.112758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:41.911860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:41.911829    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:42.257706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:42.257676    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:42.388640 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:42.388605    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:42.504278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:42.504242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:43.493316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:43.493279    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:43.533779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:43.533748    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:28:43.637027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:43.636994    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:43.770641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:43.770550    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:43.891001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:43.890966    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:45.735253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:45.735212    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:46.459309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:46.459279    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:46.528545 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:28:46.528502    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:28:46.570729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:46.570698    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:47.271755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:47.271715    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:47.571477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:47.571390    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:47.697432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:47.697401    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:49.444715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:49.444681    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:51.259497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:51.259463    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:53.304283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:53.304250    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:53.424919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:53.424799    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:54.479787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:54.479758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:54.599920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:54.599888    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:55.236409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:55.236363    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:55.396011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:55.394881    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:55.836501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:55.836468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:55.983296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:55.983252    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:56.108703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:56.108616    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:56.414282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:56.414163    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:56.714380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:56.709848    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:56.989089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:56.988976    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:57.709076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:57.709027    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:28:59.738660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:28:59.738618    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:00.955268 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:00.955232    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:01.162410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:01.162381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:02.222613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:02.222584    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:02.808771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:02.808741    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:02.972394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:02.972344    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:03.037536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.037504    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"]
Nov 02 14:29:03.038584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.038564    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy"
Nov 02 14:29:03.038637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.038587    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy"
Nov 02 14:29:03.038807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.038796    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy"
Nov 02 14:29:03.038842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.038817    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy"
Nov 02 14:29:03.043590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.043565    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.089383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.088741    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d0c9b9\""
Nov 02 14:29:03.089383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.088886    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d0c9b9-dockercfg-wzrq9\""
Nov 02 14:29:03.206906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.206861    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4lt\" (UniqueName: \"kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt\") pod \"payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.207474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.207449    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.253006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.217515    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"]
Nov 02 14:29:03.310425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.310382    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4lt\" (UniqueName: \"kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt\") pod \"payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.311545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.311511    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.312522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.312481    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.348290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.348210    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4lt\" (UniqueName: \"kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt\") pod \"payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.589706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.589675    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:03.906521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:03.906491    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"]
Nov 02 14:29:03.908542 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:29:03.908513    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bda11a_b23c_4f79_bf4d_5dfb61c66576.slice/crio-497ba0b0b9b785ef208d3ed7c7283fc777f85c175146cefc191bb8bf5d1691e5 WatchSource:0}: Error finding container 497ba0b0b9b785ef208d3ed7c7283fc777f85c175146cefc191bb8bf5d1691e5: Status 404 returned error can't find the container with id 497ba0b0b9b785ef208d3ed7c7283fc777f85c175146cefc191bb8bf5d1691e5
Nov 02 14:29:04.352133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:04.352097    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:04.887314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:04.887280    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" event={"ID":"46bda11a-b23c-4f79-bf4d-5dfb61c66576","Type":"ContainerStarted","Data":"68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801"}
Nov 02 14:29:04.887526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:04.887336    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" event={"ID":"46bda11a-b23c-4f79-bf4d-5dfb61c66576","Type":"ContainerStarted","Data":"497ba0b0b9b785ef208d3ed7c7283fc777f85c175146cefc191bb8bf5d1691e5"}
Nov 02 14:29:04.887526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:04.887409    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:04.927405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:04.927333    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" podStartSLOduration=2.927318959 podStartE2EDuration="2.927318959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:04.923679451 +0000 UTC m=+1909.467395027" watchObservedRunningTime="2026-11-02 14:29:04.927318959 +0000 UTC m=+1909.471034550"
Nov 02 14:29:05.425814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:05.425785    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:06.149060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:06.149019    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:07.580179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:07.580144    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:07.805222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:07.805186    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:08.095226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:08.095187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:08.639846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:08.639809    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:09.908147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:09.908117    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:11.716549 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:11.716514    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:13.489026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:13.488994    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:13.619458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:13.619426    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:15.264711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:15.264675    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:15.899675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:15.899636    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:29:16.542896 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:29:16.542853    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:29:17.390293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:17.390239    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:19.853817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:19.853782    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:20.021325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:20.021294    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:20.168426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:20.167333    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:20.292955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:20.292904    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:20.449382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:20.446072    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:20.648415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:20.647502    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:20.994913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:20.994868    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"]
Nov 02 14:29:21.296978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:21.296885    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:21.458923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:21.458893    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:21.988707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:21.988677    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:22.304525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:22.304430    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:22.575449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:22.575335    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:23.860939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:23.860895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:24.107643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:24.107615    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:24.976306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:24.976263    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"]
Nov 02 14:29:24.977430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:24.977403    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy"
Nov 02 14:29:24.977430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:24.977429    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da14694-3a1f-4327-aa17-77ec3cd9549f" containerName="istio-proxy"
Nov 02 14:29:24.983407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:24.983384    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.010959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.010924    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"]
Nov 02 14:29:25.015863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.015835    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f72bc6-openshift-default-dockercfg-p5hdl\""
Nov 02 14:29:25.093110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093074    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093124    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093254    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093366 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093309    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093394    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093450    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2fw\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093516    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093595    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.093645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.093637    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.194859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.194825    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195168    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195312    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195527    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2fw\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195591    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195699    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195753    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195829    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195858    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.195906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.195885    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.196221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.196152    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.196405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.196377    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.196680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.196636    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.196816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.196793    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.199238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.199212    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.199867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.199842    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.228238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.228159    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.233438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.232487    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2fw\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.280041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.280004    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:25.297856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.297816    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:25.742099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.742067    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"]
Nov 02 14:29:25.744487 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:29:25.744458    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b82088_ee55_43f7_af6c_cac45a704917.slice/crio-c54bb84ba5d37826f913c169f6635c7ffc82a1893e7a2342f2423f4f40293eb1 WatchSource:0}: Error finding container c54bb84ba5d37826f913c169f6635c7ffc82a1893e7a2342f2423f4f40293eb1: Status 404 returned error can't find the container with id c54bb84ba5d37826f913c169f6635c7ffc82a1893e7a2342f2423f4f40293eb1
Nov 02 14:29:25.747927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.747871    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:29:25.748044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.747970    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:29:25.748044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.748013    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:29:25.854028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.852010    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:25.996771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:25.996458    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q" podUID="55b61b82-9366-41ff-98e2-6930a32829d4" containerName="payload-processing" containerID="cri-o://d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b" gracePeriod=25
Nov 02 14:29:26.034290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.034247    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:26.108225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.107135    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" event={"ID":"42b82088-ee55-43f7-af6c-cac45a704917","Type":"ContainerStarted","Data":"4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262"}
Nov 02 14:29:26.108225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.107244    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" event={"ID":"42b82088-ee55-43f7-af6c-cac45a704917","Type":"ContainerStarted","Data":"c54bb84ba5d37826f913c169f6635c7ffc82a1893e7a2342f2423f4f40293eb1"}
Nov 02 14:29:26.174881 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.174064    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podStartSLOduration=2.174040189 podStartE2EDuration="2.174040189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:26.163920959 +0000 UTC m=+1930.707636552" watchObservedRunningTime="2026-11-02 14:29:26.174040189 +0000 UTC m=+1930.717755783"
Nov 02 14:29:26.300269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.298213    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:26.300484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.300271    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:26.300484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.300330    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:26.334618 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.334590    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:29:26.418299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.418264    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume\") pod \"55b61b82-9366-41ff-98e2-6930a32829d4\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") "
Nov 02 14:29:26.418299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.418314    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials\") pod \"55b61b82-9366-41ff-98e2-6930a32829d4\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") "
Nov 02 14:29:26.418604 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.418412    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume\") pod \"55b61b82-9366-41ff-98e2-6930a32829d4\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") "
Nov 02 14:29:26.418604 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.418539    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7\") pod \"55b61b82-9366-41ff-98e2-6930a32829d4\" (UID: \"55b61b82-9366-41ff-98e2-6930a32829d4\") "
Nov 02 14:29:26.418736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.418710    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume" pod "55b61b82-9366-41ff-98e2-6930a32829d4" (UID: "55b61b82-9366-41ff-98e2-6930a32829d4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:26.419104 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.419079    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume" pod "55b61b82-9366-41ff-98e2-6930a32829d4" (UID: "55b61b82-9366-41ff-98e2-6930a32829d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:26.419414 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.419394    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:29:26.419509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.419417    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b61b82-9366-41ff-98e2-6930a32829d4-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:29:26.421691 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.421663    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7" pod "55b61b82-9366-41ff-98e2-6930a32829d4" (UID: "55b61b82-9366-41ff-98e2-6930a32829d4"). InnerVolumeSpecName "kube-api-access-v2ln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:26.421797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.421751    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials" pod "55b61b82-9366-41ff-98e2-6930a32829d4" (UID: "55b61b82-9366-41ff-98e2-6930a32829d4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:26.521189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.521149    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ln7\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-kube-api-access-v2ln7\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:29:26.521189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:26.521183    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55b61b82-9366-41ff-98e2-6930a32829d4-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:29:27.115531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.115491    4492 generic.go:356] "Generic (PLEG): container finished" podID="55b61b82-9366-41ff-98e2-6930a32829d4" containerID="d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b" exitCode=0
Nov 02 14:29:27.116050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.115550    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"
Nov 02 14:29:27.116050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.115578    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q" event={"ID":"55b61b82-9366-41ff-98e2-6930a32829d4","Type":"ContainerDied","Data":"d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b"}
Nov 02 14:29:27.116050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.115645    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q" event={"ID":"55b61b82-9366-41ff-98e2-6930a32829d4","Type":"ContainerDied","Data":"2cba679900508f2ba1e58ededdc6ae6b093c1b5f318cc876c373d1d2107d8b92"}
Nov 02 14:29:27.116050 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.115677    4492 scope.go:122] "RemoveContainer" containerID="d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b"
Nov 02 14:29:27.127806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.127788    4492 scope.go:122] "RemoveContainer" containerID="d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b"
Nov 02 14:29:27.128127 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:29:27.128102    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b\": container with ID starting with d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b not found: ID does not exist" containerID="d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b"
Nov 02 14:29:27.128216 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.128134    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b"} err="failed to get container status \"d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b\": rpc error: code = NotFound desc = could not find container \"d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b\": container with ID starting with d2ff5ea5af7d073c7650997dc6f31b18b78e00cfd4fed4fc9d515e8721a2bf7b not found: ID does not exist"
Nov 02 14:29:27.171491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.171459    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"]
Nov 02 14:29:27.178517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.178488    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69gsn7q"]
Nov 02 14:29:27.298370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.298326    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:27.298555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.298416    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:27.583778 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.583740    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b61b82-9366-41ff-98e2-6930a32829d4" path="/var/lib/kubelet/pods/55b61b82-9366-41ff-98e2-6930a32829d4/volumes"
Nov 02 14:29:27.844434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:27.843957    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:28.298491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:28.298459    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:28.298899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:28.298532    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:29.298825 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:29.298784    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:29.299266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:29.298875    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:29.671816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:29.671733    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:29.798266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:29.798228    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:29.923162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:29.917776    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:30.299253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:30.299208    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:30.299716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:30.299273    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:30.890672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:30.890634    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:31.298945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:31.298909    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:31.299153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:31.298968    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:31.806978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:31.806914    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:32.298462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:32.298415    4492 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 14:29:32.298655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:32.298487    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 14:29:33.304214 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:33.304182    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:33.304767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:33.304744    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:33.305728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:33.305704    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:29:33.832411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:33.832379    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:35.806426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:35.806390    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:36.933987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:36.933948    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:37.064774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:37.064738    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:37.646509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:37.640880    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:37.876223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:37.876189    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:38.591005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.590974    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:38.846338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.846248    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"]
Nov 02 14:29:38.846989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.846973    4492 cpu_manager.go:414] "containerMap: removing container" podUID="55b61b82-9366-41ff-98e2-6930a32829d4" containerName="payload-processing"
Nov 02 14:29:38.847067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.846990    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b61b82-9366-41ff-98e2-6930a32829d4" containerName="payload-processing"
Nov 02 14:29:38.847129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.847119    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b61b82-9366-41ff-98e2-6930a32829d4" containerName="payload-processing"
Nov 02 14:29:38.850679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.850659    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:38.879398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.876278    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cmp-legacy-af7e90-provider-cred\""
Nov 02 14:29:38.879398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.876341    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-kvhrw\""
Nov 02 14:29:38.879398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.876385    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\""
Nov 02 14:29:38.879398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.876389    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\""
Nov 02 14:29:38.879398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.876408    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\""
Nov 02 14:29:38.879398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.876438    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\""
Nov 02 14:29:38.884259 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.884232    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"]
Nov 02 14:29:38.910227 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.910184    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmdk\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:38.910472 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.910319    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:38.910472 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.910435    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:38.910785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:38.910754    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.016763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.015547    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmdk\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.016763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.015679    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.016763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.015817    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.016763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.016044    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.035408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.018040    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.035675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.022895    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.035772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.022904    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.062202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.062166    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmdk\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.166060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.165951    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:39.476604 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.476502    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"]
Nov 02 14:29:39.480033 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:29:39.479989    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608977f0_911b_41f2_a4d3_eade53cea9d1.slice/crio-85b68a586130223622260e920e99c6cbd5660cedd46aba70513bede265c026c4 WatchSource:0}: Error finding container 85b68a586130223622260e920e99c6cbd5660cedd46aba70513bede265c026c4: Status 404 returned error can't find the container with id 85b68a586130223622260e920e99c6cbd5660cedd46aba70513bede265c026c4
Nov 02 14:29:39.652105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:39.652067    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:40.274490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:40.274453    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf" event={"ID":"608977f0-911b-41f2-a4d3-eade53cea9d1","Type":"ContainerStarted","Data":"825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92"}
Nov 02 14:29:40.274712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:40.274511    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf" event={"ID":"608977f0-911b-41f2-a4d3-eade53cea9d1","Type":"ContainerStarted","Data":"85b68a586130223622260e920e99c6cbd5660cedd46aba70513bede265c026c4"}
Nov 02 14:29:40.275520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:40.275482    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:40.312644 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:40.312594    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf" podStartSLOduration=2.312579224 podStartE2EDuration="2.312579224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:40.312551849 +0000 UTC m=+1944.856267441" watchObservedRunningTime="2026-11-02 14:29:40.312579224 +0000 UTC m=+1944.856294827"
Nov 02 14:29:41.435713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:41.435677    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:43.243294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:43.243254    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:45.052864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:45.052830    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:45.705595 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:45.705548    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:46.519424 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:29:46.519388    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:29:46.965237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:46.965199    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:49.036265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:49.036223    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:50.952639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:50.952596    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:52.293139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:52.293112    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:29:52.722689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:52.722653    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:54.537302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:54.537270    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:56.408908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:56.408874    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:29:58.372507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:29:58.372472    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:00.301750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.301716    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"]
Nov 02 14:30:00.307735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.307703    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"]
Nov 02 14:30:00.307906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.307846    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.312318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.312297    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.334133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.334088    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:30:00.391380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.375015    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"]
Nov 02 14:30:00.401404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.401276    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:00.428740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.428699    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"]
Nov 02 14:30:00.495461 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.495424    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.495646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.495629    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.495714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.495698    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kq2\" (UniqueName: \"kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2\") pod \"maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.495752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.495723    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzh9\" (UniqueName: \"kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9\") pod \"maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.596621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.596535    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.596775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.596638    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kq2\" (UniqueName: \"kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2\") pod \"maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.596775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.596664    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzh9\" (UniqueName: \"kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9\") pod \"maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.596775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.596735    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.597237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.597217    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.597439 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.597420    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.629263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.629233    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzh9\" (UniqueName: \"kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9\") pod \"maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.629466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.629453    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:00.629859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.629836    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kq2\" (UniqueName: \"kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2\") pod \"maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.644949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.644918    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:00.945197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.945172    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"]
Nov 02 14:30:00.947190 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:30:00.947146    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93675e01_3e86_4c68_af20_aebdca9dfc06.slice/crio-2e3daf413f6d7c3abf16dcf4f41d2a1f1dc9559e7eaa425e286eb8ce7e57dd27 WatchSource:0}: Error finding container 2e3daf413f6d7c3abf16dcf4f41d2a1f1dc9559e7eaa425e286eb8ce7e57dd27: Status 404 returned error can't find the container with id 2e3daf413f6d7c3abf16dcf4f41d2a1f1dc9559e7eaa425e286eb8ce7e57dd27
Nov 02 14:30:00.973635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:00.973027    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"]
Nov 02 14:30:00.975886 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:30:00.975837    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254fc984_ea4b_4a2e_8bbf_2185fdf0c33d.slice/crio-9c8c6144772c9c5e8dd24e03555ba6e6055fb973dfe4224a262025a32130293e WatchSource:0}: Error finding container 9c8c6144772c9c5e8dd24e03555ba6e6055fb973dfe4224a262025a32130293e: Status 404 returned error can't find the container with id 9c8c6144772c9c5e8dd24e03555ba6e6055fb973dfe4224a262025a32130293e
Nov 02 14:30:01.459092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:01.459056    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" event={"ID":"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d","Type":"ContainerStarted","Data":"80b7d791f589e863ef8270bf262d77fb73599d1d710e9d611532bd517d3d464d"}
Nov 02 14:30:01.459589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:01.459119    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" event={"ID":"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d","Type":"ContainerStarted","Data":"9c8c6144772c9c5e8dd24e03555ba6e6055fb973dfe4224a262025a32130293e"}
Nov 02 14:30:01.460853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:01.460823    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerStarted","Data":"06041e045891aa30f3b91ee905d9229d4469dccc95a1e3265b26777c953eeaca"}
Nov 02 14:30:01.461004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:01.460873    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerStarted","Data":"2e3daf413f6d7c3abf16dcf4f41d2a1f1dc9559e7eaa425e286eb8ce7e57dd27"}
Nov 02 14:30:01.500548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:01.500440    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" podStartSLOduration=1.500421274 podStartE2EDuration="1.500421274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:01.495977858 +0000 UTC m=+1966.039693451" watchObservedRunningTime="2026-11-02 14:30:01.500421274 +0000 UTC m=+1966.044136872"
Nov 02 14:30:01.535828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:01.535767    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" podStartSLOduration=1.535752897 podStartE2EDuration="1.535752897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:01.53232581 +0000 UTC m=+1966.076041524" watchObservedRunningTime="2026-11-02 14:30:01.535752897 +0000 UTC m=+1966.079468489"
Nov 02 14:30:02.448093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.448052    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:02.470417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.470383    4492 generic.go:356] "Generic (PLEG): container finished" podID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerID="80b7d791f589e863ef8270bf262d77fb73599d1d710e9d611532bd517d3d464d" exitCode=60
Nov 02 14:30:02.470890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.470488    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" event={"ID":"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d","Type":"ContainerDied","Data":"80b7d791f589e863ef8270bf262d77fb73599d1d710e9d611532bd517d3d464d"}
Nov 02 14:30:02.470963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.470885    4492 scope.go:122] "RemoveContainer" containerID="80b7d791f589e863ef8270bf262d77fb73599d1d710e9d611532bd517d3d464d"
Nov 02 14:30:02.472510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.472482    4492 generic.go:356] "Generic (PLEG): container finished" podID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerID="06041e045891aa30f3b91ee905d9229d4469dccc95a1e3265b26777c953eeaca" exitCode=60
Nov 02 14:30:02.472662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.472575    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerDied","Data":"06041e045891aa30f3b91ee905d9229d4469dccc95a1e3265b26777c953eeaca"}
Nov 02 14:30:02.472901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:02.472885    4492 scope.go:122] "RemoveContainer" containerID="06041e045891aa30f3b91ee905d9229d4469dccc95a1e3265b26777c953eeaca"
Nov 02 14:30:03.483154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:03.483116    4492 generic.go:356] "Generic (PLEG): container finished" podID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerID="68f6adb6a441a50b818a7888b5b58e8d609856ddd409840c5b12ef833707f639" exitCode=60
Nov 02 14:30:03.483632 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:03.483194    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" event={"ID":"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d","Type":"ContainerDied","Data":"68f6adb6a441a50b818a7888b5b58e8d609856ddd409840c5b12ef833707f639"}
Nov 02 14:30:03.483632 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:03.483275    4492 scope.go:122] "RemoveContainer" containerID="80b7d791f589e863ef8270bf262d77fb73599d1d710e9d611532bd517d3d464d"
Nov 02 14:30:03.483751 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:03.483647    4492 scope.go:122] "RemoveContainer" containerID="68f6adb6a441a50b818a7888b5b58e8d609856ddd409840c5b12ef833707f639"
Nov 02 14:30:03.484071 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:03.484047    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv_odh-ai-gateway-infra(254fc984-ea4b-4a2e-8bbf-2185fdf0c33d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d"
Nov 02 14:30:03.485655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:03.485633    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerStarted","Data":"eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e"}
Nov 02 14:30:04.496836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:04.496802    4492 generic.go:356] "Generic (PLEG): container finished" podID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerID="eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e" exitCode=60
Nov 02 14:30:04.497409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:04.496898    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerDied","Data":"eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e"}
Nov 02 14:30:04.497409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:04.497018    4492 scope.go:122] "RemoveContainer" containerID="06041e045891aa30f3b91ee905d9229d4469dccc95a1e3265b26777c953eeaca"
Nov 02 14:30:04.497409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:04.497282    4492 scope.go:122] "RemoveContainer" containerID="eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e"
Nov 02 14:30:04.497734 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:04.497708    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn_odh-ai-gateway-infra(93675e01-3e86-4c68-af20-aebdca9dfc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06"
Nov 02 14:30:04.499802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:04.499781    4492 scope.go:122] "RemoveContainer" containerID="68f6adb6a441a50b818a7888b5b58e8d609856ddd409840c5b12ef833707f639"
Nov 02 14:30:04.500263 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:04.500237    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv_odh-ai-gateway-infra(254fc984-ea4b-4a2e-8bbf-2185fdf0c33d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d"
Nov 02 14:30:04.528886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:04.528858    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:05.509108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:05.509079    4492 scope.go:122] "RemoveContainer" containerID="eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e"
Nov 02 14:30:05.509592 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:05.509502    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn_odh-ai-gateway-infra(93675e01-3e86-4c68-af20-aebdca9dfc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06"
Nov 02 14:30:06.338335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:06.338300    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:07.156534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:07.156498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:07.880564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:07.880532    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:08.066408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:08.066330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:08.180922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:08.180895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:08.393579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:08.393550    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:08.534063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:08.533972    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:08.803471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:08.803376    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:09.269807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:09.269770    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:10.135601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:10.135574    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:11.898416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:11.898347    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:12.025575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:12.025534    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:12.141615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:12.141579    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:13.661668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:13.661636    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:13.868728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:13.868674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:13.986860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:13.986825    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:14.248480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:14.241227    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:14.573955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:14.573834    4492 scope.go:122] "RemoveContainer" containerID="eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e"
Nov 02 14:30:14.574235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:14.574214    4492 scope.go:122] "RemoveContainer" containerID="68f6adb6a441a50b818a7888b5b58e8d609856ddd409840c5b12ef833707f639"
Nov 02 14:30:14.743539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:14.743497    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:15.245219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.245133    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:15.632608 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.632573    4492 generic.go:356] "Generic (PLEG): container finished" podID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerID="6e90600cb0d980352f254bfbef51d317ba73dc2a14b6a58f503fe6cbe4c2b249" exitCode=60
Nov 02 14:30:15.632799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.632646    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" event={"ID":"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d","Type":"ContainerDied","Data":"6e90600cb0d980352f254bfbef51d317ba73dc2a14b6a58f503fe6cbe4c2b249"}
Nov 02 14:30:15.632799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.632713    4492 scope.go:122] "RemoveContainer" containerID="68f6adb6a441a50b818a7888b5b58e8d609856ddd409840c5b12ef833707f639"
Nov 02 14:30:15.633417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.633171    4492 scope.go:122] "RemoveContainer" containerID="6e90600cb0d980352f254bfbef51d317ba73dc2a14b6a58f503fe6cbe4c2b249"
Nov 02 14:30:15.633623 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:15.633599    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv_odh-ai-gateway-infra(254fc984-ea4b-4a2e-8bbf-2185fdf0c33d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d"
Nov 02 14:30:15.635182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.635159    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerStarted","Data":"4f56f77e80beb85952d5ff3e2df1a477e9d8c9c45500cae97ec5a2e37ce2c290"}
Nov 02 14:30:15.741385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.739329    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:15.968745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:15.968697    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:16.520587 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:16.520550    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:30:16.677497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.677460    4492 generic.go:356] "Generic (PLEG): container finished" podID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerID="4f56f77e80beb85952d5ff3e2df1a477e9d8c9c45500cae97ec5a2e37ce2c290" exitCode=60
Nov 02 14:30:16.678280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.678226    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerDied","Data":"4f56f77e80beb85952d5ff3e2df1a477e9d8c9c45500cae97ec5a2e37ce2c290"}
Nov 02 14:30:16.678590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.678573    4492 scope.go:122] "RemoveContainer" containerID="eb6dfd573d4d68b45dfeb839bc8c6bcdab34e43475c7aa1aa8a64e7f48f3c93e"
Nov 02 14:30:16.679164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.679149    4492 scope.go:122] "RemoveContainer" containerID="4f56f77e80beb85952d5ff3e2df1a477e9d8c9c45500cae97ec5a2e37ce2c290"
Nov 02 14:30:16.679690 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:16.679668    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn_odh-ai-gateway-infra(93675e01-3e86-4c68-af20-aebdca9dfc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06"
Nov 02 14:30:16.731101 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.731059    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"]
Nov 02 14:30:16.781589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.781505    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"]
Nov 02 14:30:16.914841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.914810    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:16.990137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.990102    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca\") pod \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") "
Nov 02 14:30:16.990685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.990169    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2\" (UniqueName: \"kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2\") pod \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\" (UID: \"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d\") "
Nov 02 14:30:16.990685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.990575    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca" pod "254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" (UID: "254fc984-ea4b-4a2e-8bbf-2185fdf0c33d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:16.990800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.990739    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:30:16.993019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:16.992989    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2" pod "254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" (UID: "254fc984-ea4b-4a2e-8bbf-2185fdf0c33d"). InnerVolumeSpecName "kube-api-access-x9kq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:17.092029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.091925    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9kq2\" (UniqueName: \"kubernetes.io/projected/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d-kube-api-access-x9kq2\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:30:17.698841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.698808    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"
Nov 02 14:30:17.699048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.698860    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv" event={"ID":"254fc984-ea4b-4a2e-8bbf-2185fdf0c33d","Type":"ContainerDied","Data":"9c8c6144772c9c5e8dd24e03555ba6e6055fb973dfe4224a262025a32130293e"}
Nov 02 14:30:17.699048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.698924    4492 scope.go:122] "RemoveContainer" containerID="6e90600cb0d980352f254bfbef51d317ba73dc2a14b6a58f503fe6cbe4c2b249"
Nov 02 14:30:17.788901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.788289    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"]
Nov 02 14:30:17.797264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.797231    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-5a4525-29893830-tp4gv"]
Nov 02 14:30:17.856977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.856876    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:17.891152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.891121    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:17.976966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:17.976935    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:18.034079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.034038    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9\" (UniqueName: \"kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9\") pod \"93675e01-3e86-4c68-af20-aebdca9dfc06\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") "
Nov 02 14:30:18.034610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.034250    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca\") pod \"93675e01-3e86-4c68-af20-aebdca9dfc06\" (UID: \"93675e01-3e86-4c68-af20-aebdca9dfc06\") "
Nov 02 14:30:18.034684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.034640    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca" pod "93675e01-3e86-4c68-af20-aebdca9dfc06" (UID: "93675e01-3e86-4c68-af20-aebdca9dfc06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:18.034880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.034862    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93675e01-3e86-4c68-af20-aebdca9dfc06-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:30:18.036514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.036490    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9" pod "93675e01-3e86-4c68-af20-aebdca9dfc06" (UID: "93675e01-3e86-4c68-af20-aebdca9dfc06"). InnerVolumeSpecName "kube-api-access-xzzh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:18.137208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.137123    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzh9\" (UniqueName: \"kubernetes.io/projected/93675e01-3e86-4c68-af20-aebdca9dfc06-kube-api-access-xzzh9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:30:18.714387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.714317    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"
Nov 02 14:30:18.714387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.714344    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn" event={"ID":"93675e01-3e86-4c68-af20-aebdca9dfc06","Type":"ContainerDied","Data":"2e3daf413f6d7c3abf16dcf4f41d2a1f1dc9559e7eaa425e286eb8ce7e57dd27"}
Nov 02 14:30:18.714678 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.714421    4492 scope.go:122] "RemoveContainer" containerID="4f56f77e80beb85952d5ff3e2df1a477e9d8c9c45500cae97ec5a2e37ce2c290"
Nov 02 14:30:18.766486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.766441    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"]
Nov 02 14:30:18.770531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.770502    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d0c9b9-29893830-8wgzn"]
Nov 02 14:30:18.993846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:18.993758    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:19.110650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:19.110615    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:19.584917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:19.584879    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" path="/var/lib/kubelet/pods/254fc984-ea4b-4a2e-8bbf-2185fdf0c33d/volumes"
Nov 02 14:30:19.585503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:19.585473    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" path="/var/lib/kubelet/pods/93675e01-3e86-4c68-af20-aebdca9dfc06/volumes"
Nov 02 14:30:20.047555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:20.047498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:21.011569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:21.011538    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:21.129950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:21.129894    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:21.956514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:21.956479    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:22.757962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:22.757910    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:24.344378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:24.343731    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:24.475312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:24.475285    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:24.605367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:24.605252    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:25.149030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:25.148992    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:25.567331 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:25.567295    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:26.328665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:26.328623    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:26.585512 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:26.585410    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:26.909745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:26.909651    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:27.059827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:27.059257    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:28.045685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:28.045651    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:28.181511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:28.181474    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:28.323236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:28.323121    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:28.506563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:28.506512    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:29.949045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:29.948994    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:30.548058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:30.548026    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:32.348635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:32.348600    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:34.224963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:34.224930    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:36.065096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:36.065060    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:37.279879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:37.279845    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:37.746483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:37.746439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:39.425651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:39.425595    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:41.163156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:41.163124    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:41.275588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:41.275559    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:41.387710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:41.387653    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:43.044382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:43.044337    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:44.605442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:44.605405    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:44.845064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:44.845019    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:46.518204 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:30:46.518166    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:30:46.637687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:46.637654    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:48.364900 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:48.364866    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:48.495570 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:48.491923    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:48.626194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:48.626113    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:49.096505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:49.096473    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:49.272097 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:49.272046    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:49.658497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:49.658463    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:49.770041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:49.770007    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:50.333398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:50.333330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:52.155572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:52.155524    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:52.277006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:52.276968    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:54.038804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:54.038765    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:54.816233 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:54.816195    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:55.809490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:55.809463    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:30:57.624146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:30:57.624047    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:00.171367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:00.171304    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:00.426908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:00.426874    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"]
Nov 02 14:31:00.535387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:00.535079    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:00.693444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:00.690868    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:00.848379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:00.847893    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:01.624642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:01.624607    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:02.154387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:02.154324    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:02.393237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:02.393207    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:03.181845 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:03.181809    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:05.196170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.196138    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:05.427305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.427267    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf" podUID="608977f0-911b-41f2-a4d3-eade53cea9d1" containerName="payload-processing" containerID="cri-o://825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92" gracePeriod=25
Nov 02 14:31:05.445654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.445628    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:05.575965 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.575829    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:05.724946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.724870    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:31:05.898507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.898477    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume\") pod \"608977f0-911b-41f2-a4d3-eade53cea9d1\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") "
Nov 02 14:31:05.898675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.898589    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk\") pod \"608977f0-911b-41f2-a4d3-eade53cea9d1\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") "
Nov 02 14:31:05.898717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.898696    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials\") pod \"608977f0-911b-41f2-a4d3-eade53cea9d1\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") "
Nov 02 14:31:05.898758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.898715    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume\") pod \"608977f0-911b-41f2-a4d3-eade53cea9d1\" (UID: \"608977f0-911b-41f2-a4d3-eade53cea9d1\") "
Nov 02 14:31:05.898937 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.898907    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume" pod "608977f0-911b-41f2-a4d3-eade53cea9d1" (UID: "608977f0-911b-41f2-a4d3-eade53cea9d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:31:05.899146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.899128    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume" pod "608977f0-911b-41f2-a4d3-eade53cea9d1" (UID: "608977f0-911b-41f2-a4d3-eade53cea9d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:31:05.899423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.899406    4492 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-routing-overlay-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:31:05.899423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.899420    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608977f0-911b-41f2-a4d3-eade53cea9d1-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:31:05.900800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.900777    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk" pod "608977f0-911b-41f2-a4d3-eade53cea9d1" (UID: "608977f0-911b-41f2-a4d3-eade53cea9d1"). InnerVolumeSpecName "kube-api-access-qqmdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:05.900926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:05.900887    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials" pod "608977f0-911b-41f2-a4d3-eade53cea9d1" (UID: "608977f0-911b-41f2-a4d3-eade53cea9d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:06.000730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.000639    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmdk\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-kube-api-access-qqmdk\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:31:06.000730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.000675    4492 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608977f0-911b-41f2-a4d3-eade53cea9d1-provider-credentials\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:31:06.203815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.203773    4492 generic.go:356] "Generic (PLEG): container finished" podID="608977f0-911b-41f2-a4d3-eade53cea9d1" containerID="825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92" exitCode=0
Nov 02 14:31:06.204219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.203826    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"
Nov 02 14:31:06.204219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.203857    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf" event={"ID":"608977f0-911b-41f2-a4d3-eade53cea9d1","Type":"ContainerDied","Data":"825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92"}
Nov 02 14:31:06.204219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.203917    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf" event={"ID":"608977f0-911b-41f2-a4d3-eade53cea9d1","Type":"ContainerDied","Data":"85b68a586130223622260e920e99c6cbd5660cedd46aba70513bede265c026c4"}
Nov 02 14:31:06.204219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.203942    4492 scope.go:122] "RemoveContainer" containerID="825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92"
Nov 02 14:31:06.216977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.216951    4492 scope.go:122] "RemoveContainer" containerID="825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92"
Nov 02 14:31:06.217337 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:31:06.217309    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92\": container with ID starting with 825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92 not found: ID does not exist" containerID="825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92"
Nov 02 14:31:06.217456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.217347    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92"} err="failed to get container status \"825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92\": rpc error: code = NotFound desc = could not find container \"825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92\": container with ID starting with 825e392affca237a4ef1897d9f92670c918a778f452c7bf64c0244a38ac2da92 not found: ID does not exist"
Nov 02 14:31:06.252891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.252808    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"]
Nov 02 14:31:06.262418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:06.262384    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f4dwfrdf"]
Nov 02 14:31:07.230193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.230155    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:07.355874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.355825    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.555415    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"]
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556549    4492 cpu_manager.go:414] "containerMap: removing container" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556573    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556600    4492 cpu_manager.go:414] "containerMap: removing container" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556608    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556637    4492 cpu_manager.go:414] "containerMap: removing container" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556645    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556663    4492 cpu_manager.go:414] "containerMap: removing container" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556670    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556685    4492 cpu_manager.go:414] "containerMap: removing container" podUID="608977f0-911b-41f2-a4d3-eade53cea9d1" containerName="payload-processing"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556692    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608977f0-911b-41f2-a4d3-eade53cea9d1" containerName="payload-processing"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556701    4492 cpu_manager.go:414] "containerMap: removing container" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556709    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556890    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556906    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556932    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556940    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556951    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="608977f0-911b-41f2-a4d3-eade53cea9d1" containerName="payload-processing"
Nov 02 14:31:07.562438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.556960    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="254fc984-ea4b-4a2e-8bbf-2185fdf0c33d" containerName="cleanup"
Nov 02 14:31:07.563519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.562726    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.571380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.567112    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"]
Nov 02 14:31:07.618578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.604935    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e07583-dockercfg-xzlbg\""
Nov 02 14:31:07.667053 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.666289    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e07583\""
Nov 02 14:31:07.679578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.678030    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608977f0-911b-41f2-a4d3-eade53cea9d1" path="/var/lib/kubelet/pods/608977f0-911b-41f2-a4d3-eade53cea9d1/volumes"
Nov 02 14:31:07.729949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.729870    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.730154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.730096    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26f4l\" (UniqueName: \"kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l\") pod \"payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.831767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.831675    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26f4l\" (UniqueName: \"kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l\") pod \"payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.831928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.831856    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.832643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.832616    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.853775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.853748    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26f4l\" (UniqueName: \"kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l\") pod \"payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:07.968506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:07.968471    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:08.309639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.309613    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"]
Nov 02 14:31:08.312621 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:31:08.312589    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e528bec_5bc1_4638_9549_d6bbbdc486af.slice/crio-a1f4960a96e6ddaa8b977dba920b340922d34757e1735e4a7355cc109409ed5e WatchSource:0}: Error finding container a1f4960a96e6ddaa8b977dba920b340922d34757e1735e4a7355cc109409ed5e: Status 404 returned error can't find the container with id a1f4960a96e6ddaa8b977dba920b340922d34757e1735e4a7355cc109409ed5e
Nov 02 14:31:08.540505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.540467    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"]
Nov 02 14:31:08.541602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.541580    4492 cpu_manager.go:414] "containerMap: removing container" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:08.541722 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.541604    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:08.541786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.541771    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="93675e01-3e86-4c68-af20-aebdca9dfc06" containerName="cleanup"
Nov 02 14:31:08.546320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.546294    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.581714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.581618    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"]
Nov 02 14:31:08.581955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.581913    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-928a7a-openshift-default-dockercfg-864hl\""
Nov 02 14:31:08.640732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.640694    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.640918 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.640831    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.640979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.640923    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.641044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.641021    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.641131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.641112    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.641220 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.641207    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.641349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.641330    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.641421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.641393    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.641536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.641515    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ll4\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.742941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.742900    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.742949    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743013    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ll4\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743084    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743146    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743230    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743282    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743326    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743425    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743429    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.743766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.743741    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.744102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.744077    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.744390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.744322    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.744390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.744339    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.746119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.746096    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.747039 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.747019    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.765786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.765746    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.765913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.765795    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ll4\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.866234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.866130    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:08.919130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:08.919102    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:09.085378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.078841    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:09.228655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.228332    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" event={"ID":"3e528bec-5bc1-4638-9549-d6bbbdc486af","Type":"ContainerStarted","Data":"826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d"}
Nov 02 14:31:09.228655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.228419    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:09.228655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.228443    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" event={"ID":"3e528bec-5bc1-4638-9549-d6bbbdc486af","Type":"ContainerStarted","Data":"a1f4960a96e6ddaa8b977dba920b340922d34757e1735e4a7355cc109409ed5e"}
Nov 02 14:31:09.230340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.230319    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"]
Nov 02 14:31:09.232296 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:31:09.232270    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f83ca4e_b8d4_469b_ab02_58b4cc828709.slice/crio-5678fcc0fc5047966ab79f8e61ff6ef5bb040ae19179ce188bc04448a72dea46 WatchSource:0}: Error finding container 5678fcc0fc5047966ab79f8e61ff6ef5bb040ae19179ce188bc04448a72dea46: Status 404 returned error can't find the container with id 5678fcc0fc5047966ab79f8e61ff6ef5bb040ae19179ce188bc04448a72dea46
Nov 02 14:31:09.235812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.235771    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:31:09.235908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.235868    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:31:09.235947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.235909    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:31:09.274221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:09.272949    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" podStartSLOduration=2.272929224 podStartE2EDuration="2.272929224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:09.267279926 +0000 UTC m=+2033.810995517" watchObservedRunningTime="2026-11-02 14:31:09.272929224 +0000 UTC m=+2033.816644818"
Nov 02 14:31:10.243191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.243093    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" event={"ID":"0f83ca4e-b8d4-469b-ab02-58b4cc828709","Type":"ContainerStarted","Data":"d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17"}
Nov 02 14:31:10.243191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.243152    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" event={"ID":"0f83ca4e-b8d4-469b-ab02-58b4cc828709","Type":"ContainerStarted","Data":"5678fcc0fc5047966ab79f8e61ff6ef5bb040ae19179ce188bc04448a72dea46"}
Nov 02 14:31:10.314818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.314758    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podStartSLOduration=2.314745147 podStartE2EDuration="2.314745147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:10.310333517 +0000 UTC m=+2034.854049117" watchObservedRunningTime="2026-11-02 14:31:10.314745147 +0000 UTC m=+2034.858460740"
Nov 02 14:31:10.804207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.804170    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:10.879379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.878733    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:10.888420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.888284    4492 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:31:10.888420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:10.888380    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 14:31:11.867020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:11.866980    4492 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:31:11.867606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:11.867059    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 14:31:12.596435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:12.596403    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:12.869927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:12.869824    4492 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:31:12.870521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:12.870468    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 14:31:13.792446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:13.792412    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:13.869781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:13.869742    4492 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:31:13.870032 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:13.870002    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 14:31:14.465875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:14.465835    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:14.867391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:14.867280    4492 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:31:14.867589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:14.867347    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 02 14:31:15.871768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:15.871740    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:15.872250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:15.872229    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:15.872942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:15.872927    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:31:16.439946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:16.439907    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:16.579664 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:31:16.579622    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:31:18.389427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:18.389389    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:18.594119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:18.594072    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:19.005297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:19.005265    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:19.135515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:19.135484    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:20.246637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:20.246606    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:31:20.584584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:20.584500    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:22.412206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:22.412173    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:22.968274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:22.968242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:24.183955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:24.183923    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:25.148525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:25.148492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:26.095507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:26.095467    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:27.936492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:27.936421    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:28.512892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:28.512858    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:30.006448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:30.006399    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:30.757966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:30.757906    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"]
Nov 02 14:31:30.879880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:30.879845    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:31.010010 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:31.009875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:31.333743 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:31.333645    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:32.150477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:32.150437    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:33.171626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:33.171574    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:34.029812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:34.029769    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:35.492790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:35.492755    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:35.765440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:35.760951    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" podUID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" containerName="payload-pre-processing" containerID="cri-o://68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801" gracePeriod=25
Nov 02 14:31:35.909230 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:31:35.909204    4492 logging.go:55] [core] [Channel #6710 SubChannel #6711]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.133:50052", ServerName: "10.134.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.133:50052: connect: connection refused"
Nov 02 14:31:36.185611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.185518    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:36.200791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.200762    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:31:36.233072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.233041    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume\") pod \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") "
Nov 02 14:31:36.233267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.233169    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt\" (UniqueName: \"kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt\") pod \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\" (UID: \"46bda11a-b23c-4f79-bf4d-5dfb61c66576\") "
Nov 02 14:31:36.233510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.233485    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume" pod "46bda11a-b23c-4f79-bf4d-5dfb61c66576" (UID: "46bda11a-b23c-4f79-bf4d-5dfb61c66576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:31:36.233976 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.233898    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bda11a-b23c-4f79-bf4d-5dfb61c66576-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:31:36.235661 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.235635    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt" pod "46bda11a-b23c-4f79-bf4d-5dfb61c66576" (UID: "46bda11a-b23c-4f79-bf4d-5dfb61c66576"). InnerVolumeSpecName "kube-api-access-xq4lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:36.310881 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.304850    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:36.336466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.336376    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq4lt\" (UniqueName: \"kubernetes.io/projected/46bda11a-b23c-4f79-bf4d-5dfb61c66576-kube-api-access-xq4lt\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:31:36.518764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.518728    4492 generic.go:356] "Generic (PLEG): container finished" podID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" containerID="68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801" exitCode=0
Nov 02 14:31:36.519222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.518778    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"
Nov 02 14:31:36.519222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.518820    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" event={"ID":"46bda11a-b23c-4f79-bf4d-5dfb61c66576","Type":"ContainerDied","Data":"68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801"}
Nov 02 14:31:36.519222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.518876    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" event={"ID":"46bda11a-b23c-4f79-bf4d-5dfb61c66576","Type":"ContainerDied","Data":"497ba0b0b9b785ef208d3ed7c7283fc777f85c175146cefc191bb8bf5d1691e5"}
Nov 02 14:31:36.519222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.518901    4492 scope.go:122] "RemoveContainer" containerID="68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801"
Nov 02 14:31:36.532057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.532037    4492 scope.go:122] "RemoveContainer" containerID="68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801"
Nov 02 14:31:36.532454 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:31:36.532420    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801\": container with ID starting with 68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801 not found: ID does not exist" containerID="68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801"
Nov 02 14:31:36.532587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.532464    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801"} err="failed to get container status \"68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801\": rpc error: code = NotFound desc = could not find container \"68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801\": container with ID starting with 68797d8ad943c5bc85af31a602d6e6c8a1e96fdeb388bc65928f93068ab2a801 not found: ID does not exist"
Nov 02 14:31:36.554340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.554303    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:36.572250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.572220    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"]
Nov 02 14:31:36.580967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.580935    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z"]
Nov 02 14:31:36.909682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:36.909630    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0c9b9-5d6754f895-7mp4z" podUID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.133:50052\" within 1s: context deadline exceeded"
Nov 02 14:31:37.586571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:37.586537    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" path="/var/lib/kubelet/pods/46bda11a-b23c-4f79-bf4d-5dfb61c66576/volumes"
Nov 02 14:31:38.041271 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:38.041239    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:38.363173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:38.363085    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:38.506256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:38.506222    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:38.995894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:38.995861    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:39.168256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:39.168218    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:39.575128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:39.575092    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:40.133717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:40.133681    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:40.450434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:40.450386    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:41.906114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:41.906078    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:43.736825 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:43.736789    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:45.521875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:45.521841    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:46.533259 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:31:46.533219    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:31:46.565411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:46.565319    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:46.693432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:46.693395    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:47.365562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:47.365522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:47.508422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:47.508388    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:49.258099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:49.258060    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:51.041758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:51.041719    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:51.674834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:51.674746    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:52.813058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:52.813027    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:54.336255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:54.336219    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:54.627950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:54.627866    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:56.410473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:56.410432    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:58.239515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:58.239464    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:59.460083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:59.460047    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:31:59.948142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:31:59.948110    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:01.649780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:01.649750    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:03.479369 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:03.479326    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:05.345591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:05.345560    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:06.923536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:06.923503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:07.318006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:07.317883    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:09.041573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:09.041532    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:09.216630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:09.216584    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:10.943869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:10.943831    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:11.945788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:11.945757    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:12.449456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:12.449424    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:12.751129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:12.751011    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:12.962520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:12.962478    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:13.077831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:13.077747    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:13.975613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:13.975582    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:14.849302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:14.849267    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:16.524958 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:32:16.524917    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:32:16.749572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:16.749544    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:16.862581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:16.862498    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:16.986500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:16.986462    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:17.107376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:17.107327    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:18.033008 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.032979    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:18.607986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.607954    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:18.725975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.725924    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:18.845121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.845090    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:18.943199 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.943154    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"]
Nov 02 14:32:18.944001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.943977    4492 cpu_manager.go:414] "containerMap: removing container" podUID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" containerName="payload-pre-processing"
Nov 02 14:32:18.944133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.944002    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" containerName="payload-pre-processing"
Nov 02 14:32:18.944193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.944171    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bda11a-b23c-4f79-bf4d-5dfb61c66576" containerName="payload-pre-processing"
Nov 02 14:32:18.949876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.949849    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:18.970289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.970241    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"]
Nov 02 14:32:18.994124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:18.994087    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:19.032059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.024311    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5ba53f-dockercfg-9czwx\""
Nov 02 14:32:19.032059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.029563    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5ba53f\""
Nov 02 14:32:19.061518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.061474    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.105381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.077091    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjj6\" (UniqueName: \"kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6\") pod \"payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.222487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.216515    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.222487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.216597    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjj6\" (UniqueName: \"kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6\") pod \"payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.222487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.217632    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.246545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.246434    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjj6\" (UniqueName: \"kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6\") pod \"payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.273062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.272770    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.591315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.591284    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"]
Nov 02 14:32:19.947527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.947480    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" event={"ID":"e5c683bc-3642-4ce9-84e5-cb88f26e87b4","Type":"ContainerStarted","Data":"e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae"}
Nov 02 14:32:19.947729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.947703    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:19.947807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.947735    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" event={"ID":"e5c683bc-3642-4ce9-84e5-cb88f26e87b4","Type":"ContainerStarted","Data":"acfcbb532a23568fd13de419623baabdda1d4288c84570bc4e13470f1ffb0414"}
Nov 02 14:32:19.986487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:19.986411    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" podStartSLOduration=1.986349009 podStartE2EDuration="1.986349009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:19.984679126 +0000 UTC m=+2104.528394718" watchObservedRunningTime="2026-11-02 14:32:19.986349009 +0000 UTC m=+2104.530064605"
Nov 02 14:32:21.038502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:21.038469    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:22.524035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:22.524001    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:23.158667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:23.158632    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:24.821539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:24.821501    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:26.602330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:26.602214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:28.519202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:28.519168    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:30.469120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:30.469057    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:30.959275 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:30.959244    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:32:32.011917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:32.011873    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:32.170882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:32.170855    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:32.344833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:32.344743    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:32.851603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:32.851568    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:33.251342 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:33.251304    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"]
Nov 02 14:32:33.251896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:33.251609    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy" containerID="cri-o://d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17" gracePeriod=30
Nov 02 14:32:33.282404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:33.281594    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:33.823052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:33.823005    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:33.999134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:33.999097    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:34.208447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:34.208403    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:34.540913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:34.540823    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:34.681287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:34.681253    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:34.831991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:34.831900    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:35.321154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:35.321119    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:35.458787 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:35.458755    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:36.934856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:36.934817    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:37.173145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:37.173110    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:38.150857 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.150821    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:38.318903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.318844    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:38.825312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.825285    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:32:38.874044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874006    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874045    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874122    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874164    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874188    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874251    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874267    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:32:38.874526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874395    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874495    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874533    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:32:38.874757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874544    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:32:38.874757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874577    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs\") pod \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\" (UID: \"0f83ca4e-b8d4-469b-ab02-58b4cc828709\") "
Nov 02 14:32:38.874925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874802    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:38.874925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.874880    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:32:38.875477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.875451    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.875598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.875480    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.875598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.875496    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.875598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.875509    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.875598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.875520    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.876753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.876719    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "kube-api-access-95ll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:38.876847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.876789    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:32:38.876901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.876842    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:38.877003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.876989    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy" pod "0f83ca4e-b8d4-469b-ab02-58b4cc828709" (UID: "0f83ca4e-b8d4-469b-ab02-58b4cc828709"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:32:38.952028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.951941    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:38.976852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.976818    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95ll4\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-kube-api-access-95ll4\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.976852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.976855    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.977054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.976870    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:38.977054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:38.976883    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f83ca4e-b8d4-469b-ab02-58b4cc828709-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:32:39.143704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.143668    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:39.191573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.191533    4492 generic.go:356] "Generic (PLEG): container finished" podID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerID="d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17" exitCode=0
Nov 02 14:32:39.191994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.191588    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"
Nov 02 14:32:39.191994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.191616    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" event={"ID":"0f83ca4e-b8d4-469b-ab02-58b4cc828709","Type":"ContainerDied","Data":"d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17"}
Nov 02 14:32:39.191994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.191672    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb" event={"ID":"0f83ca4e-b8d4-469b-ab02-58b4cc828709","Type":"ContainerDied","Data":"5678fcc0fc5047966ab79f8e61ff6ef5bb040ae19179ce188bc04448a72dea46"}
Nov 02 14:32:39.191994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.191694    4492 scope.go:122] "RemoveContainer" containerID="d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17"
Nov 02 14:32:39.203828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.203811    4492 scope.go:122] "RemoveContainer" containerID="d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17"
Nov 02 14:32:39.204118 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:32:39.204102    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17\": container with ID starting with d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17 not found: ID does not exist" containerID="d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17"
Nov 02 14:32:39.204161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.204126    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17"} err="failed to get container status \"d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17\": rpc error: code = NotFound desc = could not find container \"d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17\": container with ID starting with d16e455b320e462b8cfb6b5e2c782c90b1a9dca8c319428710f44c884aa72a17 not found: ID does not exist"
Nov 02 14:32:39.256698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.256664    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"]
Nov 02 14:32:39.269520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.269428    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-mhgpb"]
Nov 02 14:32:39.582551 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:39.582457    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" path="/var/lib/kubelet/pods/0f83ca4e-b8d4-469b-ab02-58b4cc828709/volumes"
Nov 02 14:32:40.844899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:40.844860    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:42.609328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:42.609301    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:44.364693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:44.364618    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:46.052028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:46.051991    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:46.519247 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:32:46.519201    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:32:47.715842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.715806    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"]
Nov 02 14:32:47.716563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.716546    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy"
Nov 02 14:32:47.716563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.716563    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy"
Nov 02 14:32:47.716695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.716686    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f83ca4e-b8d4-469b-ab02-58b4cc828709" containerName="istio-proxy"
Nov 02 14:32:47.721855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.721838    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.749403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.746146    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5ba53f-serving-cert\""
Nov 02 14:32:47.749403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.748616    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"]
Nov 02 14:32:47.811685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.811644    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.811901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.811795    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.812046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.812028    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.812097 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.812080    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdmf\" (UniqueName: \"kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.848142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.848104    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:47.918410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.917381    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdmf\" (UniqueName: \"kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.918410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.917533    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.918410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.917617    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.918410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.917844    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.925549 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:32:47.925508    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-serving-cert: secret "maas-api-e2e-shared-b-w3-5ba53f-serving-cert" not found
Nov 02 14:32:47.925867 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:32:47.925847    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls podName:bfe72977-06e8-46e5-9c35-cb99090fd84a nodeName:}" failed. No retries permitted until 2026-11-02 14:32:48.425822033 +0000 UTC m=+2132.969537604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" (UID: "bfe72977-06e8-46e5-9c35-cb99090fd84a") : secret "maas-api-e2e-shared-b-w3-5ba53f-serving-cert" not found
Nov 02 14:32:47.931403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.926828    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.942118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.942078    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.964889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.964823    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdmf\" (UniqueName: \"kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:47.987747 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:47.987681    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:48.169659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:48.169620    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:48.525252 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:48.525214    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:48.528234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:48.528205    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:48.640966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:48.639590    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:49.067741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:49.067340    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"]
Nov 02 14:32:49.068136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:49.067735    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:32:49.278044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:49.278010    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" event={"ID":"bfe72977-06e8-46e5-9c35-cb99090fd84a","Type":"ContainerStarted","Data":"7b1d24a5fb9eaad791442499a7c08645e3eaf2f8da5d9fbabd3b3f777dbac787"}
Nov 02 14:32:49.931612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:49.931582    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:50.287697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:50.287613    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" event={"ID":"bfe72977-06e8-46e5-9c35-cb99090fd84a","Type":"ContainerStarted","Data":"36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3"}
Nov 02 14:32:50.288076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:50.287700    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:50.333262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:50.333205    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" podStartSLOduration=2.909069917 podStartE2EDuration="3.333190982s" totalImagesPullingTime="424.121065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:50.329265451 +0000 UTC m=+2134.872981045" watchObservedRunningTime="2026-11-02 14:32:50.333190982 +0000 UTC m=+2134.876906593"
Nov 02 14:32:51.946510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:51.946457    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:53.931449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:53.931411    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:55.618869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:55.618835    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:56.300803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:56.300777    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:32:56.649078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:56.648881    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"]
Nov 02 14:32:56.649603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:56.649166    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy" containerID="cri-o://f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28" gracePeriod=30
Nov 02 14:32:57.458652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:57.458614    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:59.329622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:59.329590    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:32:59.487840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:32:59.487807    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:01.233560 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.233505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:01.944146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.944118    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:33:01.972749 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.972716    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.972967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.972919    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.972972    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:33:01.973038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.972995    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973031    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973069    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973131    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973187    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973195    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:33:01.973297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973207    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:33:01.973297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973267    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.973550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973298    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert\") pod \"a3d1901a-51b6-4350-9c3d-4c4827d10296\" (UID: \"a3d1901a-51b6-4350-9c3d-4c4827d10296\") "
Nov 02 14:33:01.974318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.973972    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:33:01.974318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.974068    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:01.974318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.974094    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:01.974318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.974108    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:01.974318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.974280    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:01.976129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.976103    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:33:01.976129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.976121    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:33:01.976323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.976246    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:01.976323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:01.976254    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq" pod "a3d1901a-51b6-4350-9c3d-4c4827d10296" (UID: "a3d1901a-51b6-4350-9c3d-4c4827d10296"). InnerVolumeSpecName "kube-api-access-lmkdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:02.075443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.075409    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.075443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.075437    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d1901a-51b6-4350-9c3d-4c4827d10296-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.075443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.075446    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.075443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.075453    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.075818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.075460    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d1901a-51b6-4350-9c3d-4c4827d10296-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.075818 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.075470    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkdq\" (UniqueName: \"kubernetes.io/projected/a3d1901a-51b6-4350-9c3d-4c4827d10296-kube-api-access-lmkdq\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.391680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.391636    4492 generic.go:356] "Generic (PLEG): container finished" podID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerID="f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28" exitCode=0
Nov 02 14:33:02.392161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.391792    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" event={"ID":"a3d1901a-51b6-4350-9c3d-4c4827d10296","Type":"ContainerDied","Data":"f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28"}
Nov 02 14:33:02.392161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.391854    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"
Nov 02 14:33:02.392161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.391888    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh" event={"ID":"a3d1901a-51b6-4350-9c3d-4c4827d10296","Type":"ContainerDied","Data":"dec679de82cf4c845b8f84eb191f94e9baa58163ed1d1423422bf61662f247e6"}
Nov 02 14:33:02.392161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.391930    4492 scope.go:122] "RemoveContainer" containerID="f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28"
Nov 02 14:33:02.405033 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.405013    4492 scope.go:122] "RemoveContainer" containerID="f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28"
Nov 02 14:33:02.405302 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:33:02.405282    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28\": container with ID starting with f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28 not found: ID does not exist" containerID="f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28"
Nov 02 14:33:02.405391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.405315    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28"} err="failed to get container status \"f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28\": rpc error: code = NotFound desc = could not find container \"f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28\": container with ID starting with f9e5922a9cf42d7e5013c5622ac0b2571f3e41edd92655348dfa72ad52c10e28 not found: ID does not exist"
Nov 02 14:33:02.476998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.476961    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"]
Nov 02 14:33:02.485471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:02.485440    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-kk6xh"]
Nov 02 14:33:03.133259 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:03.133225    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:03.584970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:03.584930    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" path="/var/lib/kubelet/pods/a3d1901a-51b6-4350-9c3d-4c4827d10296/volumes"
Nov 02 14:33:04.995177 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:04.994654    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:06.696076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.696043    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"]
Nov 02 14:33:06.696794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.696768    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy"
Nov 02 14:33:06.696794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.696787    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy"
Nov 02 14:33:06.696951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.696938    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d1901a-51b6-4350-9c3d-4c4827d10296" containerName="istio-proxy"
Nov 02 14:33:06.702405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.702348    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.756413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.739710    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-90a96161-serving-cert\""
Nov 02 14:33:06.756413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.745248    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"]
Nov 02 14:33:06.784524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.784485    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:06.885892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.885858    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.886107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.885944    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdw8t\" (UniqueName: \"kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.886107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.886042    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.886107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.886103    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.987919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.987824    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.987919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.987887    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdw8t\" (UniqueName: \"kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.987919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.987914    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.988149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.987959    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.988710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.988640    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.990606 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.990581    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:06.990733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:06.990709    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:07.013969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:07.013935    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdw8t\" (UniqueName: \"kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t\") pod \"maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:07.041809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:07.041775    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:07.047793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:07.047765    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:07.367503 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:07.367478    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"]
Nov 02 14:33:07.369929 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:33:07.369890    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f915b26_3523_483d_b480_389acd8058b8.slice/crio-4163cb8cc13c93c6428ae835c6d5d692440ef2056a3a0097d3bb86f27810211e WatchSource:0}: Error finding container 4163cb8cc13c93c6428ae835c6d5d692440ef2056a3a0097d3bb86f27810211e: Status 404 returned error can't find the container with id 4163cb8cc13c93c6428ae835c6d5d692440ef2056a3a0097d3bb86f27810211e
Nov 02 14:33:07.430186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:07.430151    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx" event={"ID":"4f915b26-3523-483d-b480-389acd8058b8","Type":"ContainerStarted","Data":"4163cb8cc13c93c6428ae835c6d5d692440ef2056a3a0097d3bb86f27810211e"}
Nov 02 14:33:08.439085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:08.439041    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx" event={"ID":"4f915b26-3523-483d-b480-389acd8058b8","Type":"ContainerStarted","Data":"4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f"}
Nov 02 14:33:08.439725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:08.439128    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:08.481657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:08.481603    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx" podStartSLOduration=2.050212454 podStartE2EDuration="2.481587862s" totalImagesPullingTime="431.375408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:08.477147262 +0000 UTC m=+2153.020862857" watchObservedRunningTime="2026-11-02 14:33:08.481587862 +0000 UTC m=+2153.025303455"
Nov 02 14:33:08.747010 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:08.746893    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:09.173814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:09.173786    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:10.567004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:10.566915    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:12.440140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:12.440101    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:14.137186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:14.137145    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:14.411311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:14.411174    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:14.453160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:14.453132    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:33:14.687021 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:14.686988    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:14.803742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:14.803706    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:16.146147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:16.146114    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:16.524763 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:33:16.524718    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:33:17.935760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:17.935725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:18.221920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:18.221804    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:18.356789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:18.356757    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:19.788316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:19.788224    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:21.568478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:21.568440    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:23.525552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:23.525519    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:25.727687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:25.727648    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:27.744679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:27.744642    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:28.582119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:28.582082    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:29.649506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:29.649473    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:30.411793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:30.411752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:31.699521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:31.699479    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:33.497837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:33.497801    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:35.366558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:35.366522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:37.185073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:37.185034    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:38.930492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:38.930460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:39.270002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:39.269913    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:40.754583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:40.754545    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:42.710240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:42.710207    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:44.582271 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:44.582234    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:46.445947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:46.445911    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:46.553191 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:33:46.553148    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:33:48.052165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:48.052113    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:48.163247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:48.163214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:48.445379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:48.440881    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:49.419656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:49.419626    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:50.442199 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:50.442163    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:52.258381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:52.258330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:54.033652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:54.033618    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:55.215424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:55.215394    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:55.736835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:55.736804    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:57.505373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:57.505318    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:58.820510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:58.820475    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:59.388764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:59.388720    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:33:59.702757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:59.702712    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"]
Nov 02 14:33:59.703072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:33:59.703046    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy" containerID="cri-o://175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1" gracePeriod=30
Nov 02 14:34:01.130377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:01.130321    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:03.232659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:03.232627    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:04.993575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:04.993546    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:34:05.059180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.059147    4492 generic.go:356] "Generic (PLEG): container finished" podID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerID="175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1" exitCode=0
Nov 02 14:34:05.059428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.059211    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"
Nov 02 14:34:05.059428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.059207    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" event={"ID":"77a57a84-8ac8-4be1-acca-75a3319737c2","Type":"ContainerDied","Data":"175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1"}
Nov 02 14:34:05.059428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.059282    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82" event={"ID":"77a57a84-8ac8-4be1-acca-75a3319737c2","Type":"ContainerDied","Data":"83e8c8bf24358be1f3fa18154cfc8994268905e6e589b57cdc61fcff71664c0c"}
Nov 02 14:34:05.059428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.059314    4492 scope.go:122] "RemoveContainer" containerID="175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1"
Nov 02 14:34:05.077181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.077159    4492 scope.go:122] "RemoveContainer" containerID="175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1"
Nov 02 14:34:05.077750 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:34:05.077672    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1\": container with ID starting with 175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1 not found: ID does not exist" containerID="175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1"
Nov 02 14:34:05.077750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.077720    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1"} err="failed to get container status \"175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1\": rpc error: code = NotFound desc = could not find container \"175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1\": container with ID starting with 175dc014a1842183fd8ffd527d2b58f6309522997d1a377812a41f855b655ec1 not found: ID does not exist"
Nov 02 14:34:05.085936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.085879    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:05.095873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.095819    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.096338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.095941    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.096338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096089    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.096338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096127    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.096338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096261    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096428    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096449    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096532    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096570    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096630    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2\") pod \"77a57a84-8ac8-4be1-acca-75a3319737c2\" (UID: \"77a57a84-8ac8-4be1-acca-75a3319737c2\") "
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.096971    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.097394    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.097417    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.097699    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:34:05.099338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.098928    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:34:05.103392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.103320    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:34:05.103700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.103660    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:34:05.103953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.103912    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:34:05.105172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.105148    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:05.105546 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.105523    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2" pod "77a57a84-8ac8-4be1-acca-75a3319737c2" (UID: "77a57a84-8ac8-4be1-acca-75a3319737c2"). InnerVolumeSpecName "kube-api-access-pqws2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201067    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201106    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201119    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201134    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201146    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a57a84-8ac8-4be1-acca-75a3319737c2-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201159    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a57a84-8ac8-4be1-acca-75a3319737c2-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.203985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.201171    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqws2\" (UniqueName: \"kubernetes.io/projected/77a57a84-8ac8-4be1-acca-75a3319737c2-kube-api-access-pqws2\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:34:05.403412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.403372    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"]
Nov 02 14:34:05.413583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.413536    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-cdz82"]
Nov 02 14:34:05.584752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:05.584719    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" path="/var/lib/kubelet/pods/77a57a84-8ac8-4be1-acca-75a3319737c2/volumes"
Nov 02 14:34:06.412196 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:06.412158    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:06.611496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:06.611439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:07.295261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:07.295157    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:07.425846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:07.425813    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:09.307997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:09.307964    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:09.417715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:09.417675    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:11.253115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:11.253059    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:11.953141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:11.953075    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:13.028653 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:13.028610    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:14.254981 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:14.254950    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:14.812979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:14.812940    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:15.377758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:15.377724    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:15.500074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:15.500040    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:16.238948 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:16.238914    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:16.353289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:16.353256    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:16.701336 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:34:16.701281    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:34:16.702901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:16.702868    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:18.097120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:18.097084    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:18.544468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:18.544421    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:20.329115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:20.329070    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:22.148955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:22.148921    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:23.963655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:23.963616    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:25.733171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:25.733124    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:25.856095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:25.855938    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:27.516225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:27.516188    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:29.445946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:29.445910    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:31.246111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:31.246081    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:33.038872 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:33.038829    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:34.738429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:34.738398    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:34.997332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:34.997244    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:36.535689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:36.535646    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:38.093619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.093582    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"]
Nov 02 14:34:38.094295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.094280    4492 cpu_manager.go:414] "containerMap: removing container" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy"
Nov 02 14:34:38.094339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.094296    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy"
Nov 02 14:34:38.094475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.094463    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a57a84-8ac8-4be1-acca-75a3319737c2" containerName="istio-proxy"
Nov 02 14:34:38.099435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.099418    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.129059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.129026    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"]
Nov 02 14:34:38.129239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.129169    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-778b2bce-serving-cert\""
Nov 02 14:34:38.187860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.187819    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:38.191515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.191485    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.191763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.191742    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.191997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.191977    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bfhh\" (UniqueName: \"kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.192130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.192113    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.311602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.311548    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:38.313175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.313077    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.313478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.313443    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.313738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.313716    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bfhh\" (UniqueName: \"kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.313854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.313840    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.314877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.314852    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.315598 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:34:38.315581    4492 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-serving-cert: secret "maas-api-e2e-mt-778b2bce-serving-cert" not found
Nov 02 14:34:38.315786 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:34:38.315774    4492 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls podName:7bd6f7c3-c98e-411b-8ce9-7895565efc1e nodeName:}" failed. No retries permitted until 2026-11-02 14:34:38.815750979 +0000 UTC m=+2243.359466566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls") pod "maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" (UID: "7bd6f7c3-c98e-411b-8ce9-7895565efc1e") : secret "maas-api-e2e-mt-778b2bce-serving-cert" not found
Nov 02 14:34:38.319577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.319527    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.342251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.342187    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bfhh\" (UniqueName: \"kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.671645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.671610    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:38.821668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.821633    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:38.824498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:38.824466    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\") pod \"maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:39.013652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:39.013611    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:39.354309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:39.354143    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"]
Nov 02 14:34:39.357080 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:34:39.357051    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd6f7c3_c98e_411b_8ce9_7895565efc1e.slice/crio-e091859395dc1f94af59b08ed84f24ed7fad2e19e399ce96d825b88f61aef02a WatchSource:0}: Error finding container e091859395dc1f94af59b08ed84f24ed7fad2e19e399ce96d825b88f61aef02a: Status 404 returned error can't find the container with id e091859395dc1f94af59b08ed84f24ed7fad2e19e399ce96d825b88f61aef02a
Nov 02 14:34:39.401996 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:39.401961    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" event={"ID":"7bd6f7c3-c98e-411b-8ce9-7895565efc1e","Type":"ContainerStarted","Data":"e091859395dc1f94af59b08ed84f24ed7fad2e19e399ce96d825b88f61aef02a"}
Nov 02 14:34:39.407924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:39.407893    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:40.041744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:40.041652    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:40.411693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:40.411592    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" event={"ID":"7bd6f7c3-c98e-411b-8ce9-7895565efc1e","Type":"ContainerStarted","Data":"7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de"}
Nov 02 14:34:40.411693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:40.411661    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:40.480491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:40.480434    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" podStartSLOduration=2.1168206019999998 podStartE2EDuration="2.480419124s" totalImagesPullingTime="363.598522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:40.477066805 +0000 UTC m=+2245.020782399" watchObservedRunningTime="2026-11-02 14:34:40.480419124 +0000 UTC m=+2245.024134718"
Nov 02 14:34:41.951904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:41.951861    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:43.838729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:43.838694    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:45.636016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:45.635980    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:46.425919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:46.425885    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:34:46.535625 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:34:46.535574    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:34:47.406900 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:47.406870    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:49.155928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:49.155857    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:50.935910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:50.935875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:52.632687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:52.632650    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:54.385090 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:54.385051    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:56.166167 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:56.166136    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:58.046815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:58.046783    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:34:59.931641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:34:59.931599    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:01.793940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:01.793907    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:03.711144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:03.711100    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:05.621025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:05.620985    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:06.495802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:06.495763    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:06.653370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:06.653334    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:07.466281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:07.466208    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:09.242180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:09.242146    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:09.436793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:09.436760    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:11.023124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:11.023091    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"]
Nov 02 14:35:11.023713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:11.023476    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy" containerID="cri-o://4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262" gracePeriod=30
Nov 02 14:35:11.046214 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:11.046178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:12.700212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:12.700172    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:12.823386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:12.822482    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:13.014130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:13.014041    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:13.573692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:13.573651    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:14.380906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:14.380866    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:15.025837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:15.025794    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:16.331409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.331382    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:35:16.505092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505060    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505109    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505176    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505246    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505279    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505309    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505397    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505432    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505468    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy\") pod \"42b82088-ee55-43f7-af6c-cac45a704917\" (UID: \"42b82088-ee55-43f7-af6c-cac45a704917\") "
Nov 02 14:35:16.505567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505544    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:35:16.505887 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505868    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:35:16.505964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505893    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:16.506027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.505979    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:35:16.506119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.506095    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:35:16.507684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.507658    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.507808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.507690    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.507808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.507704    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.507808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.507718    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b82088-ee55-43f7-af6c-cac45a704917-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.507808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.507732    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.508710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.508681    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "kube-api-access-hk2fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:16.508869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.508849    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:35:16.509164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.509140    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:16.509698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.509674    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy" pod "42b82088-ee55-43f7-af6c-cac45a704917" (UID: "42b82088-ee55-43f7-af6c-cac45a704917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:35:16.523077 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:35:16.523025    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:35:16.609416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.609279    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2fw\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-kube-api-access-hk2fw\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.609416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.609320    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b82088-ee55-43f7-af6c-cac45a704917-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.609416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.609329    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b82088-ee55-43f7-af6c-cac45a704917-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.609416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.609337    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b82088-ee55-43f7-af6c-cac45a704917-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:35:16.797667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.797635    4492 generic.go:356] "Generic (PLEG): container finished" podID="42b82088-ee55-43f7-af6c-cac45a704917" containerID="4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262" exitCode=0
Nov 02 14:35:16.797883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.797705    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"
Nov 02 14:35:16.797883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.797713    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" event={"ID":"42b82088-ee55-43f7-af6c-cac45a704917","Type":"ContainerDied","Data":"4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262"}
Nov 02 14:35:16.797883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.797778    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m" event={"ID":"42b82088-ee55-43f7-af6c-cac45a704917","Type":"ContainerDied","Data":"c54bb84ba5d37826f913c169f6635c7ffc82a1893e7a2342f2423f4f40293eb1"}
Nov 02 14:35:16.797883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.797808    4492 scope.go:122] "RemoveContainer" containerID="4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262"
Nov 02 14:35:16.810142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.810123    4492 scope.go:122] "RemoveContainer" containerID="4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262"
Nov 02 14:35:16.810484 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:35:16.810463    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262\": container with ID starting with 4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262 not found: ID does not exist" containerID="4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262"
Nov 02 14:35:16.810568 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.810491    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262"} err="failed to get container status \"4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262\": rpc error: code = NotFound desc = could not find container \"4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262\": container with ID starting with 4c7f1bb3630d7cca554ee461f85ae80cbe003568fe607e7f6bf86f7993b6b262 not found: ID does not exist"
Nov 02 14:35:16.854866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.854826    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"]
Nov 02 14:35:16.880990 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.880725    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-kxt9m"]
Nov 02 14:35:16.933753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:16.933705    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:17.581999 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:17.581962    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b82088-ee55-43f7-af6c-cac45a704917" path="/var/lib/kubelet/pods/42b82088-ee55-43f7-af6c-cac45a704917/volumes"
Nov 02 14:35:18.750654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:18.750624    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:20.538064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:20.538029    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:22.355582 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:22.355549    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:22.539047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:22.539011    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:24.119437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:24.119400    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:25.026823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:25.026787    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:25.190936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:25.190888    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:25.729844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:25.729803    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:26.149761 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:26.149647    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:27.590581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:27.590542    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:27.953587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:27.953557    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:29.710696 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:29.710664    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:31.431898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:31.431866    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:32.486481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:32.486444    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:32.600207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:32.600161    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:32.960516 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:32.960395    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:33.251735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:33.251630    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:35.054254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:35.054202    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:36.834723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:36.834666    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:38.620741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:38.620702    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:39.353768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:39.353721    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"]
Nov 02 14:35:39.385749 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:39.385713    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"]
Nov 02 14:35:39.516841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:39.516804    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:39.638417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:39.638285    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:40.496573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:40.496534    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:40.612870 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:40.612813    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:42.437478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:42.437430    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:44.251083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:44.251036    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:44.354481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:44.354430    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" podUID="231baf73-a109-4c55-ab96-306d004fa68f" containerName="payload-processing" containerID="cri-o://68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b" gracePeriod=25
Nov 02 14:35:44.387005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:44.386969    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerName="payload-pre-processing" containerID="cri-o://978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6" gracePeriod=25
Nov 02 14:35:45.960014 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:45.959980    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:46.518434 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:35:46.518394    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:35:47.077728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:47.077686    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:47.843117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:47.843076    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:35:47.937078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:47.937045    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:48.881688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:48.881639    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" podUID="231baf73-a109-4c55-ab96-306d004fa68f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:35:48.927023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:48.926988    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:49.736681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:49.736637    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:50.739692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:50.739658    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:51.541803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:51.541755    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:53.337764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:53.337725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:55.035393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:55.035321    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:55.331702 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:55.331609    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:55.466590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:55.466322    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:56.631589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:56.631553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:56.932698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:56.932667    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:35:57.843883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:57.843844    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:35:58.881633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:58.881582    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" podUID="231baf73-a109-4c55-ab96-306d004fa68f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:35:58.966564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:35:58.966530    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:00.841643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:00.841594    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:02.637028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:02.636989    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:03.130808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:03.130775    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:04.518835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.518787    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:04.722548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.722507    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:36:04.781328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.781304    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:36:04.851117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.851082    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume\") pod \"14ac141f-2e08-452f-a3d1-54bcf6484f68\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") "
Nov 02 14:36:04.851548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.851402    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh\" (UniqueName: \"kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh\") pod \"231baf73-a109-4c55-ab96-306d004fa68f\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") "
Nov 02 14:36:04.851548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.851476    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume\") pod \"231baf73-a109-4c55-ab96-306d004fa68f\" (UID: \"231baf73-a109-4c55-ab96-306d004fa68f\") "
Nov 02 14:36:04.851548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.851468    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume" pod "14ac141f-2e08-452f-a3d1-54bcf6484f68" (UID: "14ac141f-2e08-452f-a3d1-54bcf6484f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:04.851890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.851868    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume" pod "231baf73-a109-4c55-ab96-306d004fa68f" (UID: "231baf73-a109-4c55-ab96-306d004fa68f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:04.852013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.851996    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p\" (UniqueName: \"kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p\") pod \"14ac141f-2e08-452f-a3d1-54bcf6484f68\" (UID: \"14ac141f-2e08-452f-a3d1-54bcf6484f68\") "
Nov 02 14:36:04.853009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.852987    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ac141f-2e08-452f-a3d1-54bcf6484f68-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:04.853099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.853017    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231baf73-a109-4c55-ab96-306d004fa68f-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:04.854199 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.854176    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh" pod "231baf73-a109-4c55-ab96-306d004fa68f" (UID: "231baf73-a109-4c55-ab96-306d004fa68f"). InnerVolumeSpecName "kube-api-access-jnlfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:04.854336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.854311    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p" pod "14ac141f-2e08-452f-a3d1-54bcf6484f68" (UID: "14ac141f-2e08-452f-a3d1-54bcf6484f68"). InnerVolumeSpecName "kube-api-access-69k6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:04.954810 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.954727    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlfh\" (UniqueName: \"kubernetes.io/projected/231baf73-a109-4c55-ab96-306d004fa68f-kube-api-access-jnlfh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:04.954810 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:04.954757    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69k6p\" (UniqueName: \"kubernetes.io/projected/14ac141f-2e08-452f-a3d1-54bcf6484f68-kube-api-access-69k6p\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:05.231447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.231338    4492 generic.go:356] "Generic (PLEG): container finished" podID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerID="978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6" exitCode=0
Nov 02 14:36:05.231447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.231388    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" event={"ID":"14ac141f-2e08-452f-a3d1-54bcf6484f68","Type":"ContainerDied","Data":"978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6"}
Nov 02 14:36:05.231447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.231403    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"
Nov 02 14:36:05.231447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.231448    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd" event={"ID":"14ac141f-2e08-452f-a3d1-54bcf6484f68","Type":"ContainerDied","Data":"7eb01d124663efaaa91ed992e6c9d0812a203dc030626b23d55c8c41ad551fa4"}
Nov 02 14:36:05.231788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.231471    4492 scope.go:122] "RemoveContainer" containerID="978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6"
Nov 02 14:36:05.234533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.234505    4492 generic.go:356] "Generic (PLEG): container finished" podID="231baf73-a109-4c55-ab96-306d004fa68f" containerID="68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b" exitCode=0
Nov 02 14:36:05.234687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.234553    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"
Nov 02 14:36:05.234687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.234598    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" event={"ID":"231baf73-a109-4c55-ab96-306d004fa68f","Type":"ContainerDied","Data":"68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b"}
Nov 02 14:36:05.234687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.234643    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722" event={"ID":"231baf73-a109-4c55-ab96-306d004fa68f","Type":"ContainerDied","Data":"a0c82176bf5c1cd47371e97c5a5c12e83bedab651a06d2ab88f6979d69e35536"}
Nov 02 14:36:05.245034 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.244990    4492 scope.go:122] "RemoveContainer" containerID="978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6"
Nov 02 14:36:05.245438 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:05.245411    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6\": container with ID starting with 978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6 not found: ID does not exist" containerID="978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6"
Nov 02 14:36:05.245515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.245449    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6"} err="failed to get container status \"978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6\": rpc error: code = NotFound desc = could not find container \"978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6\": container with ID starting with 978ab66c6dc5f4d0dcb1da8a8ce57edb6c847deb2680dfdea7510ec020bdeaf6 not found: ID does not exist"
Nov 02 14:36:05.245515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.245473    4492 scope.go:122] "RemoveContainer" containerID="68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b"
Nov 02 14:36:05.261922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.261901    4492 scope.go:122] "RemoveContainer" containerID="68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b"
Nov 02 14:36:05.262311 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:05.262288    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b\": container with ID starting with 68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b not found: ID does not exist" containerID="68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b"
Nov 02 14:36:05.263687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.263661    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b"} err="failed to get container status \"68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b\": rpc error: code = NotFound desc = could not find container \"68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b\": container with ID starting with 68900e1fdb36a6e547c0b189db0434a789ec6728b24b5dd336195db2bdaac54b not found: ID does not exist"
Nov 02 14:36:05.295406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.291667    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"]
Nov 02 14:36:05.298237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.298199    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9ba664-f5986c96c-4rnrd"]
Nov 02 14:36:05.324103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.324070    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"]
Nov 02 14:36:05.328494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.328463    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9ba664-54cdd94959-2r722"]
Nov 02 14:36:05.329735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.329712    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:05.446258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.446214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:05.594831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.594753    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" path="/var/lib/kubelet/pods/14ac141f-2e08-452f-a3d1-54bcf6484f68/volumes"
Nov 02 14:36:05.595203 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:05.595169    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231baf73-a109-4c55-ab96-306d004fa68f" path="/var/lib/kubelet/pods/231baf73-a109-4c55-ab96-306d004fa68f/volumes"
Nov 02 14:36:06.105473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:06.105434    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:06.418381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:06.418259    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:07.960926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:07.960897    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"]
Nov 02 14:36:07.961434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:07.961169    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" podUID="bfcc477a-aeee-4ff9-b64e-e8e10f501974" containerName="maas-api" containerID="cri-o://a60d932e03ace640ecf710dd5d27ad02b9bc412b50448ea6cb0b137d93c74ecc" gracePeriod=30
Nov 02 14:36:08.028237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.028206    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:08.315764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.315729    4492 generic.go:356] "Generic (PLEG): container finished" podID="bfcc477a-aeee-4ff9-b64e-e8e10f501974" containerID="a60d932e03ace640ecf710dd5d27ad02b9bc412b50448ea6cb0b137d93c74ecc" exitCode=0
Nov 02 14:36:08.315950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.315814    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" event={"ID":"bfcc477a-aeee-4ff9-b64e-e8e10f501974","Type":"ContainerDied","Data":"a60d932e03ace640ecf710dd5d27ad02b9bc412b50448ea6cb0b137d93c74ecc"}
Nov 02 14:36:08.407744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.407715    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:36:08.436895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.436852    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca\") pod \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") "
Nov 02 14:36:08.437110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.436966    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9\" (UniqueName: \"kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9\") pod \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") "
Nov 02 14:36:08.437110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.437015    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\") pod \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") "
Nov 02 14:36:08.437110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.437042    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs\") pod \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\" (UID: \"bfcc477a-aeee-4ff9-b64e-e8e10f501974\") "
Nov 02 14:36:08.438795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.438762    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca" pod "bfcc477a-aeee-4ff9-b64e-e8e10f501974" (UID: "bfcc477a-aeee-4ff9-b64e-e8e10f501974"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:08.444280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.444242    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls" pod "bfcc477a-aeee-4ff9-b64e-e8e10f501974" (UID: "bfcc477a-aeee-4ff9-b64e-e8e10f501974"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:08.450785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.450724    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs" pod "bfcc477a-aeee-4ff9-b64e-e8e10f501974" (UID: "bfcc477a-aeee-4ff9-b64e-e8e10f501974"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:08.452894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.452841    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9" pod "bfcc477a-aeee-4ff9-b64e-e8e10f501974" (UID: "bfcc477a-aeee-4ff9-b64e-e8e10f501974"). InnerVolumeSpecName "kube-api-access-gpws9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:08.538992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.538963    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpws9\" (UniqueName: \"kubernetes.io/projected/bfcc477a-aeee-4ff9-b64e-e8e10f501974-kube-api-access-gpws9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:08.538992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.538994    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:08.539185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.539003    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfcc477a-aeee-4ff9-b64e-e8e10f501974-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:08.539185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.539011    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfcc477a-aeee-4ff9-b64e-e8e10f501974-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:08.799573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:08.799540    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:09.324139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.324106    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"
Nov 02 14:36:09.324139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.324116    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs" event={"ID":"bfcc477a-aeee-4ff9-b64e-e8e10f501974","Type":"ContainerDied","Data":"3ced21e3ade0b77fd6f7b28a0acb57a00fed4ae16d756da71c6dc8a303c8ef6b"}
Nov 02 14:36:09.324704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.324195    4492 scope.go:122] "RemoveContainer" containerID="a60d932e03ace640ecf710dd5d27ad02b9bc412b50448ea6cb0b137d93c74ecc"
Nov 02 14:36:09.375477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.375444    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"]
Nov 02 14:36:09.384603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.384577    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9ba664-644dd4c9ff-ls5xs"]
Nov 02 14:36:09.562740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.562701    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:09.582292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:09.582221    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcc477a-aeee-4ff9-b64e-e8e10f501974" path="/var/lib/kubelet/pods/bfcc477a-aeee-4ff9-b64e-e8e10f501974/volumes"
Nov 02 14:36:10.480160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:10.480104    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:12.226419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:12.226385    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:12.347397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:12.344235    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:13.026311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.026277    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:13.136005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.135964    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:13.454877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.454843    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"]
Nov 02 14:36:13.455321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.455121    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" containerID="cri-o://44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755" gracePeriod=30
Nov 02 14:36:13.492337 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.492299    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:13.639376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.633856    4492 patch_prober.go:29] interesting pod/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:36:13.639376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:13.633934    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:36:14.129578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:14.129543    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:14.246506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:14.246477    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:15.662420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:15.662387    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"]
Nov 02 14:36:15.662900 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:15.662642    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" podUID="7bd6f7c3-c98e-411b-8ce9-7895565efc1e" containerName="maas-api" containerID="cri-o://7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de" gracePeriod=30
Nov 02 14:36:15.724955 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:15.724918    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:16.009033 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.008999    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:36:16.114985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.114886    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs\") pod \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") "
Nov 02 14:36:16.114985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.115019    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\") pod \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") "
Nov 02 14:36:16.115476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.115191    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca\") pod \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") "
Nov 02 14:36:16.115476 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.115281    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh\" (UniqueName: \"kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh\") pod \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\" (UID: \"7bd6f7c3-c98e-411b-8ce9-7895565efc1e\") "
Nov 02 14:36:16.117013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.116988    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca" pod "7bd6f7c3-c98e-411b-8ce9-7895565efc1e" (UID: "7bd6f7c3-c98e-411b-8ce9-7895565efc1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:16.120411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.120339    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs" pod "7bd6f7c3-c98e-411b-8ce9-7895565efc1e" (UID: "7bd6f7c3-c98e-411b-8ce9-7895565efc1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:16.124401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.121643    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh" pod "7bd6f7c3-c98e-411b-8ce9-7895565efc1e" (UID: "7bd6f7c3-c98e-411b-8ce9-7895565efc1e"). InnerVolumeSpecName "kube-api-access-8bfhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:16.124401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.124278    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls" pod "7bd6f7c3-c98e-411b-8ce9-7895565efc1e" (UID: "7bd6f7c3-c98e-411b-8ce9-7895565efc1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:16.219564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.219480    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:16.219564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.219516    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:16.219564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.219530    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bfhh\" (UniqueName: \"kubernetes.io/projected/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-kube-api-access-8bfhh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:16.219564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.219543    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd6f7c3-c98e-411b-8ce9-7895565efc1e-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:16.417305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.417276    4492 generic.go:356] "Generic (PLEG): container finished" podID="7bd6f7c3-c98e-411b-8ce9-7895565efc1e" containerID="7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de" exitCode=0
Nov 02 14:36:16.417533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.417336    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"
Nov 02 14:36:16.417533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.417390    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" event={"ID":"7bd6f7c3-c98e-411b-8ce9-7895565efc1e","Type":"ContainerDied","Data":"7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de"}
Nov 02 14:36:16.417533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.417447    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj" event={"ID":"7bd6f7c3-c98e-411b-8ce9-7895565efc1e","Type":"ContainerDied","Data":"e091859395dc1f94af59b08ed84f24ed7fad2e19e399ce96d825b88f61aef02a"}
Nov 02 14:36:16.417533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.417472    4492 scope.go:122] "RemoveContainer" containerID="7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de"
Nov 02 14:36:16.428932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.428907    4492 scope.go:122] "RemoveContainer" containerID="7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de"
Nov 02 14:36:16.429298 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:16.429277    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de\": container with ID starting with 7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de not found: ID does not exist" containerID="7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de"
Nov 02 14:36:16.429415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.429306    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de"} err="failed to get container status \"7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de\": rpc error: code = NotFound desc = could not find container \"7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de\": container with ID starting with 7b742ae4031a36d7eb00d54994cd4264009ae6b909f190705c2bbb388567f4de not found: ID does not exist"
Nov 02 14:36:16.452375 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.452329    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:16.465408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.465373    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"]
Nov 02 14:36:16.474755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:16.474686    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-778b2bce-6c7cbd54c5-x5wwj"]
Nov 02 14:36:16.556005 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:16.555967    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:36:17.583237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:17.583201    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd6f7c3-c98e-411b-8ce9-7895565efc1e" path="/var/lib/kubelet/pods/7bd6f7c3-c98e-411b-8ce9-7895565efc1e/volumes"
Nov 02 14:36:18.388255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.388220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:18.750826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.750798    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:36:18.861542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861504    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861603    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861649    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861712    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861797    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861844    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861886    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.861938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861923    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.862154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.861952    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket\") pod \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\" (UID: \"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3\") "
Nov 02 14:36:18.862441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.862388    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:18.862613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.862592    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.862851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.862825    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:18.863136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.863112    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:18.865334 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.865212    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:36:18.865807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.865781    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:18.868130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.867962    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "kube-api-access-z72s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:18.868229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.868207    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:18.868724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.868331    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:18.868724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.868464    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket" pod "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" (UID: "7177b655-6676-4a9f-8a9a-b9d4ab7f59c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:18.964068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964034    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964065    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z72s9\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-kube-api-access-z72s9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964075    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964083    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964095    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964103    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964110    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:18.964426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:18.964118    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:19.457190 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.457151    4492 generic.go:356] "Generic (PLEG): container finished" podID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerID="44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755" exitCode=0
Nov 02 14:36:19.457430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.457257    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"
Nov 02 14:36:19.457430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.457270    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" event={"ID":"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3","Type":"ContainerDied","Data":"44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755"}
Nov 02 14:36:19.457430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.457335    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx" event={"ID":"7177b655-6676-4a9f-8a9a-b9d4ab7f59c3","Type":"ContainerDied","Data":"1cefa9da1c674498bd070feeed6328bfab5978bfb1b1421866ed7feca04f3644"}
Nov 02 14:36:19.457430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.457403    4492 scope.go:122] "RemoveContainer" containerID="44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755"
Nov 02 14:36:19.469114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.469088    4492 scope.go:122] "RemoveContainer" containerID="961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"
Nov 02 14:36:19.479755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.479732    4492 scope.go:122] "RemoveContainer" containerID="44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755"
Nov 02 14:36:19.480087 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:19.480060    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755\": container with ID starting with 44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755 not found: ID does not exist" containerID="44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755"
Nov 02 14:36:19.480163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.480098    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755"} err="failed to get container status \"44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755\": rpc error: code = NotFound desc = could not find container \"44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755\": container with ID starting with 44c5928f3ba70c7e2958cdb173539f86639ac563c2a20a19c126bd922a17d755 not found: ID does not exist"
Nov 02 14:36:19.480163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.480138    4492 scope.go:122] "RemoveContainer" containerID="961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"
Nov 02 14:36:19.480444 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:19.480421    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237\": container with ID starting with 961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237 not found: ID does not exist" containerID="961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"
Nov 02 14:36:19.480510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.480452    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237"} err="failed to get container status \"961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237\": rpc error: code = NotFound desc = could not find container \"961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237\": container with ID starting with 961e6f93555605d3376edf8bc9428c455f53131f8f51e1fd8679240a5571a237 not found: ID does not exist"
Nov 02 14:36:19.517561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.517529    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"]
Nov 02 14:36:19.527509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.527480    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9ba664-openshift-default-5b57547dc7-sgdhx"]
Nov 02 14:36:19.584288 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:19.584255    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" path="/var/lib/kubelet/pods/7177b655-6676-4a9f-8a9a-b9d4ab7f59c3/volumes"
Nov 02 14:36:20.003407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.001698    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"]
Nov 02 14:36:20.003407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.001978    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx" podUID="4f915b26-3523-483d-b480-389acd8058b8" containerName="maas-api" containerID="cri-o://4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f" gracePeriod=30
Nov 02 14:36:20.092504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.092466    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:20.290188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.290166    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:36:20.393380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.386117    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca\") pod \"4f915b26-3523-483d-b480-389acd8058b8\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") "
Nov 02 14:36:20.393380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.386195    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls\") pod \"4f915b26-3523-483d-b480-389acd8058b8\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") "
Nov 02 14:36:20.393380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.386291    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs\") pod \"4f915b26-3523-483d-b480-389acd8058b8\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") "
Nov 02 14:36:20.393380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.386454    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t\" (UniqueName: \"kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t\") pod \"4f915b26-3523-483d-b480-389acd8058b8\" (UID: \"4f915b26-3523-483d-b480-389acd8058b8\") "
Nov 02 14:36:20.393380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.391063    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca" pod "4f915b26-3523-483d-b480-389acd8058b8" (UID: "4f915b26-3523-483d-b480-389acd8058b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:20.435850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.435609    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls" pod "4f915b26-3523-483d-b480-389acd8058b8" (UID: "4f915b26-3523-483d-b480-389acd8058b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:20.442809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.442507    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs" pod "4f915b26-3523-483d-b480-389acd8058b8" (UID: "4f915b26-3523-483d-b480-389acd8058b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:20.442809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.442554    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t" pod "4f915b26-3523-483d-b480-389acd8058b8" (UID: "4f915b26-3523-483d-b480-389acd8058b8"). InnerVolumeSpecName "kube-api-access-mdw8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:20.511429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.493826    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f915b26-3523-483d-b480-389acd8058b8-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:20.511429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.493877    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:20.511429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.493892    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f915b26-3523-483d-b480-389acd8058b8-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:20.511429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.493905    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdw8t\" (UniqueName: \"kubernetes.io/projected/4f915b26-3523-483d-b480-389acd8058b8-kube-api-access-mdw8t\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:20.568393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.566775    4492 generic.go:356] "Generic (PLEG): container finished" podID="4f915b26-3523-483d-b480-389acd8058b8" containerID="4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f" exitCode=0
Nov 02 14:36:20.568393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.566948    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx" event={"ID":"4f915b26-3523-483d-b480-389acd8058b8","Type":"ContainerDied","Data":"4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f"}
Nov 02 14:36:20.568393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.566995    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx" event={"ID":"4f915b26-3523-483d-b480-389acd8058b8","Type":"ContainerDied","Data":"4163cb8cc13c93c6428ae835c6d5d692440ef2056a3a0097d3bb86f27810211e"}
Nov 02 14:36:20.568393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.567021    4492 scope.go:122] "RemoveContainer" containerID="4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f"
Nov 02 14:36:20.568393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.567130    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"
Nov 02 14:36:20.583231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.583216    4492 scope.go:122] "RemoveContainer" containerID="4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f"
Nov 02 14:36:20.583611 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:20.583591    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f\": container with ID starting with 4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f not found: ID does not exist" containerID="4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f"
Nov 02 14:36:20.583685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.583619    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f"} err="failed to get container status \"4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f\": rpc error: code = NotFound desc = could not find container \"4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f\": container with ID starting with 4af92b47eb92db95e33b762b427cab1aa9e799dbccedb8cac2d89e3312c9eb2f not found: ID does not exist"
Nov 02 14:36:20.620988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.620959    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"]
Nov 02 14:36:20.630168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.630142    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90a96161-5bf6dbc7b-nfvvx"]
Nov 02 14:36:20.859971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:20.859849    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:21.259949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:21.259882    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:21.389671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:21.389611    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:21.583026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:21.582931    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f915b26-3523-483d-b480-389acd8058b8" path="/var/lib/kubelet/pods/4f915b26-3523-483d-b480-389acd8058b8/volumes"
Nov 02 14:36:21.942928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:21.942891    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"]
Nov 02 14:36:21.979709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:21.979666    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"]
Nov 02 14:36:22.087687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:22.087638    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:22.217434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:22.212752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:22.597869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:22.597788    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:23.082752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:23.082724    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:23.216156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:23.216125    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:25.041857 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:25.041815    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:26.713393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:26.713336    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"]
Nov 02 14:36:26.714291 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:26.714257    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" podUID="a2ea0d2a-a00d-4644-bf30-ba64ef76d052" containerName="maas-api" containerID="cri-o://e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1" gracePeriod=30
Nov 02 14:36:26.772095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:26.772062    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:26.943960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:26.943922    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" podUID="00fd286f-4693-4acc-9087-588159f7198d" containerName="payload-processing" containerID="cri-o://677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26" gracePeriod=25
Nov 02 14:36:26.988316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:26.984444    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerName="payload-pre-processing" containerID="cri-o://34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164" gracePeriod=25
Nov 02 14:36:27.055833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.055806    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:36:27.162903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.162860    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh\" (UniqueName: \"kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh\") pod \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") "
Nov 02 14:36:27.163082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.162983    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs\") pod \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") "
Nov 02 14:36:27.163082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.163024    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca\") pod \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") "
Nov 02 14:36:27.163082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.163062    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\") pod \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\" (UID: \"a2ea0d2a-a00d-4644-bf30-ba64ef76d052\") "
Nov 02 14:36:27.164088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.164056    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca" pod "a2ea0d2a-a00d-4644-bf30-ba64ef76d052" (UID: "a2ea0d2a-a00d-4644-bf30-ba64ef76d052"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:27.166465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.166437    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls" pod "a2ea0d2a-a00d-4644-bf30-ba64ef76d052" (UID: "a2ea0d2a-a00d-4644-bf30-ba64ef76d052"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:27.166976 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.166954    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs" pod "a2ea0d2a-a00d-4644-bf30-ba64ef76d052" (UID: "a2ea0d2a-a00d-4644-bf30-ba64ef76d052"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:27.177426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.177403    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh" pod "a2ea0d2a-a00d-4644-bf30-ba64ef76d052" (UID: "a2ea0d2a-a00d-4644-bf30-ba64ef76d052"). InnerVolumeSpecName "kube-api-access-q6gwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:27.269431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.265016    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6gwh\" (UniqueName: \"kubernetes.io/projected/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-kube-api-access-q6gwh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:27.269431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.265050    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:27.269431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.265065    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:27.269431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.265074    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ea0d2a-a00d-4644-bf30-ba64ef76d052-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:27.458740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.458709    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:27.626730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.626697    4492 generic.go:356] "Generic (PLEG): container finished" podID="a2ea0d2a-a00d-4644-bf30-ba64ef76d052" containerID="e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1" exitCode=0
Nov 02 14:36:27.626923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.626746    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"
Nov 02 14:36:27.626923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.626784    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" event={"ID":"a2ea0d2a-a00d-4644-bf30-ba64ef76d052","Type":"ContainerDied","Data":"e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1"}
Nov 02 14:36:27.626923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.626845    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x" event={"ID":"a2ea0d2a-a00d-4644-bf30-ba64ef76d052","Type":"ContainerDied","Data":"2427dcaca10d1024580b487d4dacb25c6b7d7547d85f890696d4e4924b33aa80"}
Nov 02 14:36:27.626923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.626873    4492 scope.go:122] "RemoveContainer" containerID="e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1"
Nov 02 14:36:27.637660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.637634    4492 scope.go:122] "RemoveContainer" containerID="e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1"
Nov 02 14:36:27.637972 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:27.637952    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1\": container with ID starting with e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1 not found: ID does not exist" containerID="e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1"
Nov 02 14:36:27.638017 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.637981    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1"} err="failed to get container status \"e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1\": rpc error: code = NotFound desc = could not find container \"e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1\": container with ID starting with e65711347b723f2a24065c6a078a825118329145cf201c03559ab506950532b1 not found: ID does not exist"
Nov 02 14:36:27.678180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.678129    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"]
Nov 02 14:36:27.681632 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:27.681605    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12b8da-6558df86d-q2r9x"]
Nov 02 14:36:29.296402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:29.296337    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:29.585731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:29.585656    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ea0d2a-a00d-4644-bf30-ba64ef76d052" path="/var/lib/kubelet/pods/a2ea0d2a-a00d-4644-bf30-ba64ef76d052/volumes"
Nov 02 14:36:29.705295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:29.705266    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:29.815869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:29.815839    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:30.459810 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:30.459778    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:30.782431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:30.782301    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"]
Nov 02 14:36:30.782618 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:30.782568    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy" containerID="cri-o://9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e" gracePeriod=30
Nov 02 14:36:30.811308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:30.811279    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:31.066737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:31.066631    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:31.178493 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:31.178456    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:31.296114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:31.296083    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:31.413297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:31.413206    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:31.868397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:31.868318    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:32.342163 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:32.342125    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:32.813637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:32.813605    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:32.966603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:32.966570    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:33.187107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:33.187078    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:33.303821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:33.303784    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:34.522675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:34.522638    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:34.632509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:34.632477    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:34.796007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:34.795906    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:35.120274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:35.120181    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" podUID="00fd286f-4693-4acc-9087-588159f7198d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:36:35.157814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:35.157779    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:35.803786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:35.803753    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:35.917109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:35.917083    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:36.030707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.030665    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:36.105306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.105281    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:36:36.129293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.129258    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:36:36.197808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.197775    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.197829    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.197875    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.197925    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.197999    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198064    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198090    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198104    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:36.198170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198152    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198245    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data\") pod \"c7b90407-cb9e-400b-a369-e04b59cde7cc\" (UID: \"c7b90407-cb9e-400b-a369-e04b59cde7cc\") "
Nov 02 14:36:36.198435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198393    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:36.198496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198476    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:36.198681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198656    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:36.198872 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.198847    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:36.199067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.199031    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.199067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.199057    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.199067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.199073    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.199067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.199086    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.199067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.199096    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b90407-cb9e-400b-a369-e04b59cde7cc-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.200957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.200927    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:36.201071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.201000    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:36:36.201196 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.201172    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:36:36.201548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.201524    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg" pod "c7b90407-cb9e-400b-a369-e04b59cde7cc" (UID: "c7b90407-cb9e-400b-a369-e04b59cde7cc"). InnerVolumeSpecName "kube-api-access-cfwwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:36.300072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.300035    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwwg\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-kube-api-access-cfwwg\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.300072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.300069    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.300072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.300078    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.300300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.300086    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b90407-cb9e-400b-a369-e04b59cde7cc-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:36.415709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.415621    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:36.705184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.705149    4492 generic.go:356] "Generic (PLEG): container finished" podID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerID="9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e" exitCode=0
Nov 02 14:36:36.705396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.705197    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"
Nov 02 14:36:36.705396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.705229    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" event={"ID":"c7b90407-cb9e-400b-a369-e04b59cde7cc","Type":"ContainerDied","Data":"9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e"}
Nov 02 14:36:36.705396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.705286    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh" event={"ID":"c7b90407-cb9e-400b-a369-e04b59cde7cc","Type":"ContainerDied","Data":"945e507beca65be92dc2a9032fdb20d8b9b602df5130a5210a2db4c5c9f977ef"}
Nov 02 14:36:36.705396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.705309    4492 scope.go:122] "RemoveContainer" containerID="9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e"
Nov 02 14:36:36.718420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.718399    4492 scope.go:122] "RemoveContainer" containerID="99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"
Nov 02 14:36:36.731645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.731621    4492 scope.go:122] "RemoveContainer" containerID="9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e"
Nov 02 14:36:36.732013 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:36.731989    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e\": container with ID starting with 9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e not found: ID does not exist" containerID="9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e"
Nov 02 14:36:36.732105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.732026    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e"} err="failed to get container status \"9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e\": rpc error: code = NotFound desc = could not find container \"9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e\": container with ID starting with 9c2c7fa31e439290cd95411969475c6de554e09cd63bd96eeffa13a48685293e not found: ID does not exist"
Nov 02 14:36:36.732105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.732049    4492 scope.go:122] "RemoveContainer" containerID="99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"
Nov 02 14:36:36.732349 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:36.732331    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05\": container with ID starting with 99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05 not found: ID does not exist" containerID="99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"
Nov 02 14:36:36.732349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.732380    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05"} err="failed to get container status \"99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05\": rpc error: code = NotFound desc = could not find container \"99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05\": container with ID starting with 99812e1fb13d3d769b0900157d5219c0e61c88134e7eca46f4548a3a04582d05 not found: ID does not exist"
Nov 02 14:36:36.760277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.760235    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"]
Nov 02 14:36:36.770510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:36.770472    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-n47dh"]
Nov 02 14:36:37.126928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:37.126842    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:37.263561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:37.263525    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:37.583124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:37.583084    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" path="/var/lib/kubelet/pods/c7b90407-cb9e-400b-a369-e04b59cde7cc/volumes"
Nov 02 14:36:39.052451 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:39.052400    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:39.706803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:39.706770    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:40.840636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:40.840594    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:40.977563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:40.977346    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:41.111731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:41.111650    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:41.474545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:41.474504    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:42.763656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:42.763624    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:43.957716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:43.957684    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"]
Nov 02 14:36:43.998798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:43.998760    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"]
Nov 02 14:36:44.105682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:44.105644    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:44.222774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:44.222681    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:44.694417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:44.694370    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:45.046294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:45.046213    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:45.120750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:45.120717    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" podUID="00fd286f-4693-4acc-9087-588159f7198d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:36:46.129260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:46.129217    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:36:46.518844 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:46.518802    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:36:46.541222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:46.541188    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:47.240601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.240569    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:36:47.266462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.266434    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:36:47.362742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.362715    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume\") pod \"00fd286f-4693-4acc-9087-588159f7198d\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") "
Nov 02 14:36:47.362937 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.362756    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume\") pod \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") "
Nov 02 14:36:47.362937 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.362842    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm\" (UniqueName: \"kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm\") pod \"00fd286f-4693-4acc-9087-588159f7198d\" (UID: \"00fd286f-4693-4acc-9087-588159f7198d\") "
Nov 02 14:36:47.363027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.362984    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl\" (UniqueName: \"kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl\") pod \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\" (UID: \"47119e4f-f07a-4c80-a49e-dd9bc05c9659\") "
Nov 02 14:36:47.363445 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.363185    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume" pod "00fd286f-4693-4acc-9087-588159f7198d" (UID: "00fd286f-4693-4acc-9087-588159f7198d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:47.363445 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.363209    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume" pod "47119e4f-f07a-4c80-a49e-dd9bc05c9659" (UID: "47119e4f-f07a-4c80-a49e-dd9bc05c9659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:47.363643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.363526    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fd286f-4693-4acc-9087-588159f7198d-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:47.363643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.363545    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47119e4f-f07a-4c80-a49e-dd9bc05c9659-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:47.365443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.365416    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm" pod "00fd286f-4693-4acc-9087-588159f7198d" (UID: "00fd286f-4693-4acc-9087-588159f7198d"). InnerVolumeSpecName "kube-api-access-ncnpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:47.365567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.365498    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl" pod "47119e4f-f07a-4c80-a49e-dd9bc05c9659" (UID: "47119e4f-f07a-4c80-a49e-dd9bc05c9659"). InnerVolumeSpecName "kube-api-access-dj5sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:47.464471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.464334    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj5sl\" (UniqueName: \"kubernetes.io/projected/47119e4f-f07a-4c80-a49e-dd9bc05c9659-kube-api-access-dj5sl\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:47.464471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.464411    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnpm\" (UniqueName: \"kubernetes.io/projected/00fd286f-4693-4acc-9087-588159f7198d-kube-api-access-ncnpm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:36:47.794244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.794149    4492 generic.go:356] "Generic (PLEG): container finished" podID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerID="34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164" exitCode=0
Nov 02 14:36:47.794244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.794189    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" event={"ID":"47119e4f-f07a-4c80-a49e-dd9bc05c9659","Type":"ContainerDied","Data":"34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164"}
Nov 02 14:36:47.794244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.794228    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"
Nov 02 14:36:47.794550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.794253    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw" event={"ID":"47119e4f-f07a-4c80-a49e-dd9bc05c9659","Type":"ContainerDied","Data":"2b313b038abb6edd0595cea723a14a7934808c23dd36170430dd26ae5c8ab11e"}
Nov 02 14:36:47.794550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.794279    4492 scope.go:122] "RemoveContainer" containerID="34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164"
Nov 02 14:36:47.796194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.796171    4492 generic.go:356] "Generic (PLEG): container finished" podID="00fd286f-4693-4acc-9087-588159f7198d" containerID="677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26" exitCode=0
Nov 02 14:36:47.796275 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.796235    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" event={"ID":"00fd286f-4693-4acc-9087-588159f7198d","Type":"ContainerDied","Data":"677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26"}
Nov 02 14:36:47.796325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.796279    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l" event={"ID":"00fd286f-4693-4acc-9087-588159f7198d","Type":"ContainerDied","Data":"f43a7d2e59b7ca7de293b46bb9040877bd6566949e9cd0b664f8d7c76902858c"}
Nov 02 14:36:47.796394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.796375    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"
Nov 02 14:36:47.806405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.806339    4492 scope.go:122] "RemoveContainer" containerID="34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164"
Nov 02 14:36:47.806747 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:47.806721    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164\": container with ID starting with 34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164 not found: ID does not exist" containerID="34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164"
Nov 02 14:36:47.806819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.806764    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164"} err="failed to get container status \"34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164\": rpc error: code = NotFound desc = could not find container \"34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164\": container with ID starting with 34b924f638698902d038aa453b0bfe7019d1c3d80781e3a6b50310f51a0e7164 not found: ID does not exist"
Nov 02 14:36:47.806819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.806792    4492 scope.go:122] "RemoveContainer" containerID="677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26"
Nov 02 14:36:47.818740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.818714    4492 scope.go:122] "RemoveContainer" containerID="677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26"
Nov 02 14:36:47.819090 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:36:47.819067    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26\": container with ID starting with 677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26 not found: ID does not exist" containerID="677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26"
Nov 02 14:36:47.819155 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.819103    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26"} err="failed to get container status \"677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26\": rpc error: code = NotFound desc = could not find container \"677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26\": container with ID starting with 677d1251b7953b86f89d22ccecbc1f30560338f848aeb236a1f232721eea4f26 not found: ID does not exist"
Nov 02 14:36:47.843513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.843469    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"]
Nov 02 14:36:47.850051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.850015    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-12b8da-84bc4c4dc5-2lwjw"]
Nov 02 14:36:47.880790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.880732    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"]
Nov 02 14:36:47.885115 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:47.885073    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12b8da-7496c4f78f-2xs4l"]
Nov 02 14:36:48.027095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:48.027056    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:48.465573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:48.465527    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:48.958779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:48.958705    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerName="payload-processing" containerID="cri-o://d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8" gracePeriod=25
Nov 02 14:36:48.999748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:48.999699    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerName="payload-pre-processing" containerID="cri-o://3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d" gracePeriod=25
Nov 02 14:36:49.434915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:49.434833    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:49.549302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:49.549266    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:49.583330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:49.583297    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fd286f-4693-4acc-9087-588159f7198d" path="/var/lib/kubelet/pods/00fd286f-4693-4acc-9087-588159f7198d/volumes"
Nov 02 14:36:49.583875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:49.583851    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" path="/var/lib/kubelet/pods/47119e4f-f07a-4c80-a49e-dd9bc05c9659/volumes"
Nov 02 14:36:50.419055 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:50.419017    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:51.599616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:51.599574    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:52.186932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:52.186903    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:52.439520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:52.439407    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:54.021084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:54.021052    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:54.653563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:54.653509    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:55.328231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:55.328195    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:36:55.328695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:55.328193    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:36:55.588889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:55.588788    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:56.230732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:56.230623    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:58.104297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:58.104261    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:36:59.906547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:36:59.906492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:00.038175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:00.038086    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:01.850067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:01.850016    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:03.970478 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:03.970436    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:05.327988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:05.327952    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:37:05.327988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:05.327973    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:37:05.728671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:05.728639    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:07.526749 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:07.526712    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:09.250315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.250241    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:09.349918 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.349887    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:37:09.353772 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.353744    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:37:09.452270 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.452223    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume\") pod \"fbeee856-ad31-49c8-a84b-d9f3588a8127\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") "
Nov 02 14:37:09.452520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.452326    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps\" (UniqueName: \"kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps\") pod \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") "
Nov 02 14:37:09.452520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.452410    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8\" (UniqueName: \"kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8\") pod \"fbeee856-ad31-49c8-a84b-d9f3588a8127\" (UID: \"fbeee856-ad31-49c8-a84b-d9f3588a8127\") "
Nov 02 14:37:09.452520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.452468    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume\") pod \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\" (UID: \"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576\") "
Nov 02 14:37:09.452765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.452737    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume" pod "fbeee856-ad31-49c8-a84b-d9f3588a8127" (UID: "fbeee856-ad31-49c8-a84b-d9f3588a8127"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:09.453099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.453063    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume" pod "c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" (UID: "c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:09.453254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.453191    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbeee856-ad31-49c8-a84b-d9f3588a8127-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:09.455302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.455270    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8" pod "fbeee856-ad31-49c8-a84b-d9f3588a8127" (UID: "fbeee856-ad31-49c8-a84b-d9f3588a8127"). InnerVolumeSpecName "kube-api-access-w96f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:09.455463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.455438    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps" pod "c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" (UID: "c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576"). InnerVolumeSpecName "kube-api-access-sxlps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:09.554921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.554805    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlps\" (UniqueName: \"kubernetes.io/projected/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-kube-api-access-sxlps\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:09.554921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.554842    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96f8\" (UniqueName: \"kubernetes.io/projected/fbeee856-ad31-49c8-a84b-d9f3588a8127-kube-api-access-w96f8\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:09.554921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.554851    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:09.696843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:09.696811    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:10.003284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.003244    4492 generic.go:356] "Generic (PLEG): container finished" podID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerID="3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d" exitCode=0
Nov 02 14:37:10.003484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.003299    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"
Nov 02 14:37:10.003484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.003337    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" event={"ID":"fbeee856-ad31-49c8-a84b-d9f3588a8127","Type":"ContainerDied","Data":"3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d"}
Nov 02 14:37:10.003484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.003435    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp" event={"ID":"fbeee856-ad31-49c8-a84b-d9f3588a8127","Type":"ContainerDied","Data":"e503a84ec52f5efda6e778e39ae499b3af3b85676296ded0bdd028205a0a6194"}
Nov 02 14:37:10.003484 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.003458    4492 scope.go:122] "RemoveContainer" containerID="3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d"
Nov 02 14:37:10.004930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.004903    4492 generic.go:356] "Generic (PLEG): container finished" podID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerID="d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8" exitCode=0
Nov 02 14:37:10.005051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.004968    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"
Nov 02 14:37:10.005098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.004967    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" event={"ID":"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576","Type":"ContainerDied","Data":"d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8"}
Nov 02 14:37:10.005207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.005190    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx" event={"ID":"c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576","Type":"ContainerDied","Data":"3ed15f3b0f0aa2673f10db6f413768c9a77124cc5f7709d7b8f40d90768d41d5"}
Nov 02 14:37:10.014673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.014651    4492 scope.go:122] "RemoveContainer" containerID="3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d"
Nov 02 14:37:10.015031 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:37:10.015005    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d\": container with ID starting with 3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d not found: ID does not exist" containerID="3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d"
Nov 02 14:37:10.015084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.015040    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d"} err="failed to get container status \"3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d\": rpc error: code = NotFound desc = could not find container \"3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d\": container with ID starting with 3c88f2728162b53a17e29668cbe576d1a2f4365988a20fd055e1dfef8a7b856d not found: ID does not exist"
Nov 02 14:37:10.015084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.015060    4492 scope.go:122] "RemoveContainer" containerID="d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8"
Nov 02 14:37:10.025428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.025405    4492 scope.go:122] "RemoveContainer" containerID="d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8"
Nov 02 14:37:10.025786 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:37:10.025764    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8\": container with ID starting with d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8 not found: ID does not exist" containerID="d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8"
Nov 02 14:37:10.025852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.025795    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8"} err="failed to get container status \"d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8\": rpc error: code = NotFound desc = could not find container \"d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8\": container with ID starting with d6300ace278f1d183c77395660b1d025702d933c272ad53d0d671090572d28b8 not found: ID does not exist"
Nov 02 14:37:10.053249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.053190    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"]
Nov 02 14:37:10.064419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.064381    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4be645-544cb69f75-b77xx"]
Nov 02 14:37:10.089555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.089521    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"]
Nov 02 14:37:10.092991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:10.092962    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-4be645-58d4569f87-754xp"]
Nov 02 14:37:11.042910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.042870    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:11.581588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.581558    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" path="/var/lib/kubelet/pods/c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576/volumes"
Nov 02 14:37:11.581997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.581983    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" path="/var/lib/kubelet/pods/fbeee856-ad31-49c8-a84b-d9f3588a8127/volumes"
Nov 02 14:37:11.742328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742285    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"]
Nov 02 14:37:11.742931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742915    4492 cpu_manager.go:414] "containerMap: removing container" podUID="231baf73-a109-4c55-ab96-306d004fa68f" containerName="payload-processing"
Nov 02 14:37:11.742980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742933    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231baf73-a109-4c55-ab96-306d004fa68f" containerName="payload-processing"
Nov 02 14:37:11.742980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742949    4492 cpu_manager.go:414] "containerMap: removing container" podUID="7bd6f7c3-c98e-411b-8ce9-7895565efc1e" containerName="maas-api"
Nov 02 14:37:11.742980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742954    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd6f7c3-c98e-411b-8ce9-7895565efc1e" containerName="maas-api"
Nov 02 14:37:11.742980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742968    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a2ea0d2a-a00d-4644-bf30-ba64ef76d052" containerName="maas-api"
Nov 02 14:37:11.742980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742973    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ea0d2a-a00d-4644-bf30-ba64ef76d052" containerName="maas-api"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742985    4492 cpu_manager.go:414] "containerMap: removing container" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.742990    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743000    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerName="payload-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743004    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerName="payload-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743018    4492 cpu_manager.go:414] "containerMap: removing container" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerName="payload-pre-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743023    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerName="payload-pre-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743031    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743035    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743041    4492 cpu_manager.go:414] "containerMap: removing container" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerName="payload-pre-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743046    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerName="payload-pre-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743055    4492 cpu_manager.go:414] "containerMap: removing container" podUID="00fd286f-4693-4acc-9087-588159f7198d" containerName="payload-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743059    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fd286f-4693-4acc-9087-588159f7198d" containerName="payload-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743068    4492 cpu_manager.go:414] "containerMap: removing container" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743073    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743080    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4f915b26-3523-483d-b480-389acd8058b8" containerName="maas-api"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743085    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f915b26-3523-483d-b480-389acd8058b8" containerName="maas-api"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743095    4492 cpu_manager.go:414] "containerMap: removing container" podUID="bfcc477a-aeee-4ff9-b64e-e8e10f501974" containerName="maas-api"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743099    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcc477a-aeee-4ff9-b64e-e8e10f501974" containerName="maas-api"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743108    4492 cpu_manager.go:414] "containerMap: removing container" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerName="payload-pre-processing"
Nov 02 14:37:11.743144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743112    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerName="payload-pre-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743190    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f915b26-3523-483d-b480-389acd8058b8" containerName="maas-api"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743198    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcc477a-aeee-4ff9-b64e-e8e10f501974" containerName="maas-api"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743204    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ac141f-2e08-452f-a3d1-54bcf6484f68" containerName="payload-pre-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743210    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="231baf73-a109-4c55-ab96-306d004fa68f" containerName="payload-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743216    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd6f7c3-c98e-411b-8ce9-7895565efc1e" containerName="maas-api"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743224    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ea0d2a-a00d-4644-bf30-ba64ef76d052" containerName="maas-api"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743232    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743238    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c413edf3-9fa9-43aa-9e8a-0c8c8e6bb576" containerName="payload-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743245    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeee856-ad31-49c8-a84b-d9f3588a8127" containerName="payload-pre-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743252    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743256    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="47119e4f-f07a-4c80-a49e-dd9bc05c9659" containerName="payload-pre-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743261    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fd286f-4693-4acc-9087-588159f7198d" containerName="payload-processing"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743266    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b82088-ee55-43f7-af6c-cac45a704917" containerName="istio-proxy"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743270    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy"
Nov 02 14:37:11.743740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.743276    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy"
Nov 02 14:37:11.747326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.747280    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.774656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.774620    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"]
Nov 02 14:37:11.886592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.886487    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.886592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.886543    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1f3047-805a-4404-9b58-57ffe5f98e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.886843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.886649    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.886843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.886711    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfnd\" (UniqueName: \"kubernetes.io/projected/9a1f3047-805a-4404-9b58-57ffe5f98e65-kube-api-access-cqfnd\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.886957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.886854    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.886957 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.886907    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.887055 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.887025    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.887055 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.887052    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.887165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.887135    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.988901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.988862    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.988911    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.988951    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.988978    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989117 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989007    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989123    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989153    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1f3047-805a-4404-9b58-57ffe5f98e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989183    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989223    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfnd\" (UniqueName: \"kubernetes.io/projected/9a1f3047-805a-4404-9b58-57ffe5f98e65-kube-api-access-cqfnd\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989403    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989519    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989626    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.989720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989678    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.990001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.989937    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a1f3047-805a-4404-9b58-57ffe5f98e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.991645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.991620    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:11.991876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:11.991856    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:12.013371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.013323    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfnd\" (UniqueName: \"kubernetes.io/projected/9a1f3047-805a-4404-9b58-57ffe5f98e65-kube-api-access-cqfnd\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:12.013371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.013338    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a1f3047-805a-4404-9b58-57ffe5f98e65-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-4q677\" (UID: \"9a1f3047-805a-4404-9b58-57ffe5f98e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:12.067336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.067302    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:12.393004 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.392969    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"]
Nov 02 14:37:12.395921 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:37:12.395881    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1f3047_805a_4404_9b58_57ffe5f98e65.slice/crio-1da2ff3786caa47f68579209dbaa0e151d57da4c0e64324b46aa5b2da6d15be5 WatchSource:0}: Error finding container 1da2ff3786caa47f68579209dbaa0e151d57da4c0e64324b46aa5b2da6d15be5: Status 404 returned error can't find the container with id 1da2ff3786caa47f68579209dbaa0e151d57da4c0e64324b46aa5b2da6d15be5
Nov 02 14:37:12.398581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.398545    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:37:12.398679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.398615    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:37:12.398679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.398644    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:37:12.999534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:12.999491    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:13.035744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.034583    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" event={"ID":"9a1f3047-805a-4404-9b58-57ffe5f98e65","Type":"ContainerStarted","Data":"c7b2b43c2c0177c8a695b55e8fccc89d40a5487746c305a236cc9c62d6fe9385"}
Nov 02 14:37:13.035744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.034658    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" event={"ID":"9a1f3047-805a-4404-9b58-57ffe5f98e65","Type":"ContainerStarted","Data":"1da2ff3786caa47f68579209dbaa0e151d57da4c0e64324b46aa5b2da6d15be5"}
Nov 02 14:37:13.070868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.070836    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:13.072321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.072290    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:13.072486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.072398    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:13.090200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.090140    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podStartSLOduration=2.090123556 podStartE2EDuration="2.090123556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:13.085903559 +0000 UTC m=+2397.629619153" watchObservedRunningTime="2026-11-02 14:37:13.090123556 +0000 UTC m=+2397.633839149"
Nov 02 14:37:13.189379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.188992    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:13.486110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.486075    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:13.868890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:13.868798    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:14.068373 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:14.068318    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:14.068545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:14.068393    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:14.281851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:14.281809    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:15.068593 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:15.068557    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:15.068777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:15.068624    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:15.721761 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:15.721729    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:16.068562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:16.068475    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:16.068562 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:16.068530    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:16.520185 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:37:16.520143    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:37:16.930485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:16.930438    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:17.067918 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:17.067883    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:17.068102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:17.067938    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:18.067766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:18.067717    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:18.068266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:18.067800    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:18.279605 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:18.279565    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:19.068406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:19.068347    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 14:37:19.068960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:19.068452    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 14:37:19.515221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:19.515178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:20.227964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:20.227915    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:20.939594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:20.939562    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:20.940531 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:20.940506    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:20.941230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:20.941210    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677"
Nov 02 14:37:22.043019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:22.042974    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:23.594207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:23.594166    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:23.749923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:23.749888    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:23.865135 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:23.865052    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:24.142121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:24.142027    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:24.487316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:24.487279    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"]
Nov 02 14:37:24.487587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:24.487560    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy" containerID="cri-o://6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c" gracePeriod=30
Nov 02 14:37:24.519974 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:24.519878    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:25.907346 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:25.907307    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:27.655916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:27.655871    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:29.395660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.395583    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:29.779889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.779868    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:37:29.810497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810460    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810514    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810545    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810571    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810611    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810718    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810766    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:37:29.810906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810839    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810885    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.810906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810897    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:37:29.811176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810914    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:37:29.811176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.810936    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594\") pod \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\" (UID: \"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d\") "
Nov 02 14:37:29.811176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811111    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:37:29.811176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811118    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:29.811804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811765    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.811804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811797    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.811993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811813    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.811993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811827    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.811993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.811840    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.813149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.813120    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:29.813257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.813220    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "kube-api-access-sn594". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:29.813301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.813251    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:37:29.813301 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.813288    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo" pod "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" (UID: "cf3d5ac5-3b5d-4e4f-917c-e44c137c453d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:37:29.913448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.913323    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.913448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.913393    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.913448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.913405    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn594\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-kube-api-access-sn594\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:29.913448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:29.913415    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:37:30.190907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.190874    4492 generic.go:356] "Generic (PLEG): container finished" podID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerID="6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c" exitCode=0
Nov 02 14:37:30.191106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.190924    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"
Nov 02 14:37:30.191106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.190965    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" event={"ID":"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d","Type":"ContainerDied","Data":"6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c"}
Nov 02 14:37:30.191106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.191043    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl" event={"ID":"cf3d5ac5-3b5d-4e4f-917c-e44c137c453d","Type":"ContainerDied","Data":"d9b05cc5bebbcdf2ccfc10c06df11e74c6c894e8037d24ba9ad72f1a04727422"}
Nov 02 14:37:30.191106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.191066    4492 scope.go:122] "RemoveContainer" containerID="6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c"
Nov 02 14:37:30.202258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.202237    4492 scope.go:122] "RemoveContainer" containerID="6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c"
Nov 02 14:37:30.202572 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:37:30.202553    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c\": container with ID starting with 6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c not found: ID does not exist" containerID="6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c"
Nov 02 14:37:30.202652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.202583    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c"} err="failed to get container status \"6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c\": rpc error: code = NotFound desc = could not find container \"6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c\": container with ID starting with 6569b6b650b36619972d3ca703520d7e36dc4941601f289c11ad5cc68d14e99c not found: ID does not exist"
Nov 02 14:37:30.239997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.239962    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"]
Nov 02 14:37:30.243343 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:30.243317    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfjnl"]
Nov 02 14:37:31.263225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:31.263193    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:31.582039 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:31.581942    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" path="/var/lib/kubelet/pods/cf3d5ac5-3b5d-4e4f-917c-e44c137c453d/volumes"
Nov 02 14:37:33.383378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:33.383257    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:34.957590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:34.957546    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:37:34.958297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:34.958242    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:37:34.974823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:34.974789    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:37:34.975065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:34.975032    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:37:35.043578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:35.043533    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:37:35.043894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:35.043857    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:37:35.448554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:35.448504    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:37.260475 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:37.260439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:39.134276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:39.134241    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:39.799570 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:39.799523    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:40.822548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:40.822516    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:42.610704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:42.610605    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:44.323277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:44.323220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:46.213717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:46.213496    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:46.517264 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:37:46.517162    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:37:48.042862 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:48.042814    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:49.775132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:49.775087    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:51.596728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:51.596691    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:53.305713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:53.305668    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:55.163486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:55.163452    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:57.140870 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:57.140828    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:37:58.957318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:37:58.957279    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:00.580336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:00.580297    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:00.877242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:00.877147    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:02.837310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:02.837274    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:04.631049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:04.631020    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:06.428440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:06.428402    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:08.238578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:08.238536    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:09.839036 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:09.839004    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:10.143916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:10.143834    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:12.129762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:12.129726    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:14.059730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:14.056870    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:15.824109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:15.824041    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:16.519231 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:38:16.519193    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:38:17.547596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:17.547553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:19.095756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:19.095721    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:19.457389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:19.457315    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:19.678740 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:19.678700    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:21.430543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:21.430505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:23.224783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:23.224751    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:25.022472 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:25.022437    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:26.429022 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:26.428890    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:26.965708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:26.965670    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:27.078884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:27.078849    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:28.730942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:28.730905    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:29.245132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:29.245100    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:30.725100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:30.725063    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:31.993863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:31.993819    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:32.496176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:32.496142    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:34.281461 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:34.281422    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:36.173040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:36.173000    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:36.896435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:36.896396    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:38.104901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:38.104865    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:39.554806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:39.554763    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:39.730302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:39.730266    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"]
Nov 02 14:38:39.761035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:39.761004    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"]
Nov 02 14:38:39.845982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:39.845871    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:39.968159 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:39.968111    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:41.725551 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:41.725506    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:43.807259 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:43.807203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:43.980446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:43.980410    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:44.731466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:44.731374    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerName="payload-processing" containerID="cri-o://fcb5fb841c22375c53d31f83c934163386267dd181aabb16a496989cd33cddc9" gracePeriod=25
Nov 02 14:38:44.762040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:44.761989    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerName="payload-pre-processing" containerID="cri-o://ce3eedc4c4b2537f73d5d12fd1fc6a5f2e44ad57546ffb2aab1f724b87db452a" gracePeriod=25
Nov 02 14:38:45.529413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:45.529369    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:46.296172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:46.296141    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:46.517986 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:38:46.517879    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:38:47.318694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:47.318660    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:49.082802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:49.082765    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:49.197592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:49.197555    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:49.534189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:49.534147    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:49.644748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:49.644713    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:49.844468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:49.844343    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:38:49.844468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:49.844426    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:38:51.001378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:51.001313    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:53.002396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:53.002337    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:54.836242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:54.836203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:56.547636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:56.547590    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:58.038505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.038474    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"]
Nov 02 14:38:58.039283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.039249    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2" podUID="329d2121-eaad-4b40-af4c-31ec54cd398b" containerName="maas-api" containerID="cri-o://e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc" gracePeriod=30
Nov 02 14:38:58.111989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.111958    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:58.391244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.391208    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:38:58.432116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.432088    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs\") pod \"329d2121-eaad-4b40-af4c-31ec54cd398b\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") "
Nov 02 14:38:58.432116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.432126    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4\" (UniqueName: \"kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4\") pod \"329d2121-eaad-4b40-af4c-31ec54cd398b\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") "
Nov 02 14:38:58.432446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.432196    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca\") pod \"329d2121-eaad-4b40-af4c-31ec54cd398b\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") "
Nov 02 14:38:58.432446 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.432237    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls\") pod \"329d2121-eaad-4b40-af4c-31ec54cd398b\" (UID: \"329d2121-eaad-4b40-af4c-31ec54cd398b\") "
Nov 02 14:38:58.433418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.432631    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca" pod "329d2121-eaad-4b40-af4c-31ec54cd398b" (UID: "329d2121-eaad-4b40-af4c-31ec54cd398b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:38:58.433418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.432732    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329d2121-eaad-4b40-af4c-31ec54cd398b-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:38:58.435802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.435508    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls" pod "329d2121-eaad-4b40-af4c-31ec54cd398b" (UID: "329d2121-eaad-4b40-af4c-31ec54cd398b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:38:58.435802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.435796    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4" pod "329d2121-eaad-4b40-af4c-31ec54cd398b" (UID: "329d2121-eaad-4b40-af4c-31ec54cd398b"). InnerVolumeSpecName "kube-api-access-5mbn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:38:58.442210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.442166    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs" pod "329d2121-eaad-4b40-af4c-31ec54cd398b" (UID: "329d2121-eaad-4b40-af4c-31ec54cd398b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:38:58.552756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.552674    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:38:58.552756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.552714    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329d2121-eaad-4b40-af4c-31ec54cd398b-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:38:58.552756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.552728    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mbn4\" (UniqueName: \"kubernetes.io/projected/329d2121-eaad-4b40-af4c-31ec54cd398b-kube-api-access-5mbn4\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:38:58.856993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.856926    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:38:58.947647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.947603    4492 generic.go:356] "Generic (PLEG): container finished" podID="329d2121-eaad-4b40-af4c-31ec54cd398b" containerID="e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc" exitCode=0
Nov 02 14:38:58.947821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.947654    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"
Nov 02 14:38:58.947821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.947687    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2" event={"ID":"329d2121-eaad-4b40-af4c-31ec54cd398b","Type":"ContainerDied","Data":"e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc"}
Nov 02 14:38:58.947821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.947741    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2" event={"ID":"329d2121-eaad-4b40-af4c-31ec54cd398b","Type":"ContainerDied","Data":"6bc168fa48429cc1b49da3d6690fa45a6c494787c9dbac132194d95b942e3e0b"}
Nov 02 14:38:58.947821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.947762    4492 scope.go:122] "RemoveContainer" containerID="e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc"
Nov 02 14:38:58.958151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.958136    4492 scope.go:122] "RemoveContainer" containerID="e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc"
Nov 02 14:38:58.958455 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:38:58.958435    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc\": container with ID starting with e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc not found: ID does not exist" containerID="e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc"
Nov 02 14:38:58.958509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.958463    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc"} err="failed to get container status \"e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc\": rpc error: code = NotFound desc = could not find container \"e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc\": container with ID starting with e2b0123203d23f909eb77a74d2ac1d88b3b58e974ef619985e2a4d32ae1556dc not found: ID does not exist"
Nov 02 14:38:58.995083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:58.995048    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"]
Nov 02 14:38:59.003949 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:59.003901    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f1e1c2-dc4dddc45-6j8r2"]
Nov 02 14:38:59.581263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:59.581229    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329d2121-eaad-4b40-af4c-31ec54cd398b" path="/var/lib/kubelet/pods/329d2121-eaad-4b40-af4c-31ec54cd398b/volumes"
Nov 02 14:38:59.844225 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:59.844121    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:38:59.844921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:38:59.844880    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:39:00.574960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:00.574919    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:01.765108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:01.765077    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:02.199035 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:02.199002    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:02.315101 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:02.315063    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:03.005365 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:03.005327    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:03.113313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:03.113275    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:03.754951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:03.754919    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:04.029157 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:04.029079    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:04.344875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:04.344793    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:04.997311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:04.997082    4492 generic.go:356] "Generic (PLEG): container finished" podID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerID="ce3eedc4c4b2537f73d5d12fd1fc6a5f2e44ad57546ffb2aab1f724b87db452a" exitCode=0
Nov 02 14:39:04.997311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:04.997254    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" event={"ID":"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b","Type":"ContainerDied","Data":"ce3eedc4c4b2537f73d5d12fd1fc6a5f2e44ad57546ffb2aab1f724b87db452a"}
Nov 02 14:39:05.002898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.002744    4492 generic.go:356] "Generic (PLEG): container finished" podID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerID="fcb5fb841c22375c53d31f83c934163386267dd181aabb16a496989cd33cddc9" exitCode=0
Nov 02 14:39:05.002898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.002811    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" event={"ID":"7181ff04-9a1e-40c8-b261-81ffcc373eef","Type":"ContainerDied","Data":"fcb5fb841c22375c53d31f83c934163386267dd181aabb16a496989cd33cddc9"}
Nov 02 14:39:05.058575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.058541    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:39:05.087318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.087295    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:39:05.147614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.147578    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx\" (UniqueName: \"kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx\") pod \"7181ff04-9a1e-40c8-b261-81ffcc373eef\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") "
Nov 02 14:39:05.147813 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.147648    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume\") pod \"7181ff04-9a1e-40c8-b261-81ffcc373eef\" (UID: \"7181ff04-9a1e-40c8-b261-81ffcc373eef\") "
Nov 02 14:39:05.149724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.149397    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume" pod "7181ff04-9a1e-40c8-b261-81ffcc373eef" (UID: "7181ff04-9a1e-40c8-b261-81ffcc373eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:39:05.167885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.167848    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx" pod "7181ff04-9a1e-40c8-b261-81ffcc373eef" (UID: "7181ff04-9a1e-40c8-b261-81ffcc373eef"). InnerVolumeSpecName "kube-api-access-7h6cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:05.176782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.176748    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:05.249630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.249538    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh\" (UniqueName: \"kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh\") pod \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") "
Nov 02 14:39:05.249630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.249584    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume\") pod \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\" (UID: \"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b\") "
Nov 02 14:39:05.250056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.250023    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume" pod "9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" (UID: "9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:39:05.250278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.250247    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6cx\" (UniqueName: \"kubernetes.io/projected/7181ff04-9a1e-40c8-b261-81ffcc373eef-kube-api-access-7h6cx\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:05.250278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.250266    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7181ff04-9a1e-40c8-b261-81ffcc373eef-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:05.250278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.250275    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:05.251965 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.251934    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh" pod "9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" (UID: "9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b"). InnerVolumeSpecName "kube-api-access-5vngh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:05.352207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.352163    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vngh\" (UniqueName: \"kubernetes.io/projected/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b-kube-api-access-5vngh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:05.896502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:05.896393    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:06.010020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.009989    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"
Nov 02 14:39:06.010020 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.010011    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t" event={"ID":"7181ff04-9a1e-40c8-b261-81ffcc373eef","Type":"ContainerDied","Data":"81b2606cde8567219bb7208d69d9c5d2ff982af8698b42174391b877cdcc2b74"}
Nov 02 14:39:06.010262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.010072    4492 scope.go:122] "RemoveContainer" containerID="fcb5fb841c22375c53d31f83c934163386267dd181aabb16a496989cd33cddc9"
Nov 02 14:39:06.011754 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.011703    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"
Nov 02 14:39:06.012107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.011715    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr" event={"ID":"9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b","Type":"ContainerDied","Data":"9d63ddf6a6d2aee01527f954ff19def49b6e464b5043b18d44584ff2537d9dee"}
Nov 02 14:39:06.020499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.020480    4492 scope.go:122] "RemoveContainer" containerID="ce3eedc4c4b2537f73d5d12fd1fc6a5f2e44ad57546ffb2aab1f724b87db452a"
Nov 02 14:39:06.063577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.063533    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"]
Nov 02 14:39:06.072916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.072885    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e1c2-bf5f7dfff-zdmfr"]
Nov 02 14:39:06.097124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.097085    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"]
Nov 02 14:39:06.104945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:06.104913    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e1c2-6db8c44df7-2788t"]
Nov 02 14:39:07.035323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:07.035292    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"]
Nov 02 14:39:07.144904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:07.144873    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:07.267329 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:07.267288    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:07.582659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:07.582628    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" path="/var/lib/kubelet/pods/7181ff04-9a1e-40c8-b261-81ffcc373eef/volumes"
Nov 02 14:39:07.583095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:07.583080    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" path="/var/lib/kubelet/pods/9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b/volumes"
Nov 02 14:39:07.734380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:07.734278    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:09.092329 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:09.092265    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:09.633322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:09.633278    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:09.943417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:09.943335    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:11.054576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:11.054543    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:11.199130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:11.199096    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:11.535583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:11.535546    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:12.036640 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:12.036578    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" containerName="payload-pre-processing" containerID="cri-o://91397fe69ff0eb20433d2ffebc5068c9ef9d7d90850c9aadceb56a66e1b689e3" gracePeriod=25
Nov 02 14:39:12.944622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:12.944567    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:13.316573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:13.316490    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:39:13.347986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:13.347946    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:15.040059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:15.040005    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:15.675710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:15.675666    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:16.518600 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:39:16.518567    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:39:16.835739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:16.835648    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:18.647284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:18.647251    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:19.864384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:19.864328    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:20.530564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:20.530530    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:21.037965 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:21.037923    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:22.130836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:22.130804    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:22.878802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:22.878769    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:22.993591 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:22.993558    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:23.316003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:23.315966    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:39:24.603246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:24.603213    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:26.046341 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.046300    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"]
Nov 02 14:39:26.047556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.047483    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" podUID="de86f7d7-e096-429a-96d7-828c085393c9" containerName="maas-api" containerID="cri-o://ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359" gracePeriod=30
Nov 02 14:39:26.104253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.104218    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:26.411875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.411845    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:39:26.594175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.594145    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\") pod \"de86f7d7-e096-429a-96d7-828c085393c9\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") "
Nov 02 14:39:26.594368 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.594236    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca\") pod \"de86f7d7-e096-429a-96d7-828c085393c9\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") "
Nov 02 14:39:26.594368 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.594259    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs\") pod \"de86f7d7-e096-429a-96d7-828c085393c9\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") "
Nov 02 14:39:26.594467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.594392    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf\" (UniqueName: \"kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf\") pod \"de86f7d7-e096-429a-96d7-828c085393c9\" (UID: \"de86f7d7-e096-429a-96d7-828c085393c9\") "
Nov 02 14:39:26.594705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.594672    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca" pod "de86f7d7-e096-429a-96d7-828c085393c9" (UID: "de86f7d7-e096-429a-96d7-828c085393c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:39:26.594807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.594784    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86f7d7-e096-429a-96d7-828c085393c9-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:26.596653 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.596627    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs" pod "de86f7d7-e096-429a-96d7-828c085393c9" (UID: "de86f7d7-e096-429a-96d7-828c085393c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:39:26.596785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.596660    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf" pod "de86f7d7-e096-429a-96d7-828c085393c9" (UID: "de86f7d7-e096-429a-96d7-828c085393c9"). InnerVolumeSpecName "kube-api-access-mtktf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:26.596785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.596682    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls" pod "de86f7d7-e096-429a-96d7-828c085393c9" (UID: "de86f7d7-e096-429a-96d7-828c085393c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:39:26.695846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.695803    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtktf\" (UniqueName: \"kubernetes.io/projected/de86f7d7-e096-429a-96d7-828c085393c9-kube-api-access-mtktf\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:26.695846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.695840    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:26.695846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.695855    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86f7d7-e096-429a-96d7-828c085393c9-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:26.767267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.767234    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:26.886300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:26.886262    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:27.186166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.186131    4492 generic.go:356] "Generic (PLEG): container finished" podID="de86f7d7-e096-429a-96d7-828c085393c9" containerID="ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359" exitCode=0
Nov 02 14:39:27.186657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.186181    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"
Nov 02 14:39:27.186657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.186198    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" event={"ID":"de86f7d7-e096-429a-96d7-828c085393c9","Type":"ContainerDied","Data":"ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359"}
Nov 02 14:39:27.186657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.186253    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp" event={"ID":"de86f7d7-e096-429a-96d7-828c085393c9","Type":"ContainerDied","Data":"20a9ff2188b009cf8ef4e464e9b1b40da4b1621c38104766615d9a7480d588f9"}
Nov 02 14:39:27.186657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.186275    4492 scope.go:122] "RemoveContainer" containerID="ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359"
Nov 02 14:39:27.198173 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.198154    4492 scope.go:122] "RemoveContainer" containerID="ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359"
Nov 02 14:39:27.198532 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:39:27.198511    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359\": container with ID starting with ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359 not found: ID does not exist" containerID="ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359"
Nov 02 14:39:27.198622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.198541    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359"} err="failed to get container status \"ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359\": rpc error: code = NotFound desc = could not find container \"ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359\": container with ID starting with ebb469a0bfb214bbe00084e9dcf23a8baa12f4900e9469a0b390f8dfb6f61359 not found: ID does not exist"
Nov 02 14:39:27.241670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.241625    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"]
Nov 02 14:39:27.251566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.251534    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-623b4b-7c4b96987c-87lxp"]
Nov 02 14:39:27.582246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:27.582150    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de86f7d7-e096-429a-96d7-828c085393c9" path="/var/lib/kubelet/pods/de86f7d7-e096-429a-96d7-828c085393c9/volumes"
Nov 02 14:39:28.379899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:28.379863    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:28.612078 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:28.612017    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:29.001842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:29.001803    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:29.110480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:29.110443    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:29.811128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:29.811083    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:30.213574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:30.213518    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"]
Nov 02 14:39:30.213854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:30.213719    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy" containerID="cri-o://ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878" gracePeriod=30
Nov 02 14:39:30.252635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:30.252602    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:30.638843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:30.625590    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:30.770754 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:30.770719    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:31.525558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:31.525520    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:31.914482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:31.914393    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"]
Nov 02 14:39:31.914713 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:31.914678    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy" containerID="cri-o://4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78" gracePeriod=30
Nov 02 14:39:31.962306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:31.962268    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:32.232746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.232714    4492 generic.go:356] "Generic (PLEG): container finished" podID="b907231f-8cb0-4bed-86e5-898b4145d948" containerID="91397fe69ff0eb20433d2ffebc5068c9ef9d7d90850c9aadceb56a66e1b689e3" exitCode=0
Nov 02 14:39:32.232964 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.232945    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" event={"ID":"b907231f-8cb0-4bed-86e5-898b4145d948","Type":"ContainerDied","Data":"91397fe69ff0eb20433d2ffebc5068c9ef9d7d90850c9aadceb56a66e1b689e3"}
Nov 02 14:39:32.313076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.313049    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:39:32.494501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.494416    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm\" (UniqueName: \"kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm\") pod \"b907231f-8cb0-4bed-86e5-898b4145d948\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") "
Nov 02 14:39:32.494659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.494570    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume\") pod \"b907231f-8cb0-4bed-86e5-898b4145d948\" (UID: \"b907231f-8cb0-4bed-86e5-898b4145d948\") "
Nov 02 14:39:32.495001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.494965    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume" pod "b907231f-8cb0-4bed-86e5-898b4145d948" (UID: "b907231f-8cb0-4bed-86e5-898b4145d948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:39:32.496726 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.496709    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm" pod "b907231f-8cb0-4bed-86e5-898b4145d948" (UID: "b907231f-8cb0-4bed-86e5-898b4145d948"). InnerVolumeSpecName "kube-api-access-z9knm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:32.595891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.595862    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b907231f-8cb0-4bed-86e5-898b4145d948-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:32.595891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.595888    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9knm\" (UniqueName: \"kubernetes.io/projected/b907231f-8cb0-4bed-86e5-898b4145d948-kube-api-access-z9knm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:32.725468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:32.725428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:33.243136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:33.243107    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"
Nov 02 14:39:33.243312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:33.243104    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk" event={"ID":"b907231f-8cb0-4bed-86e5-898b4145d948","Type":"ContainerDied","Data":"b22cc349d8b157762750ffdff1e8adfb642e9d019140620a2fbb5c11c095dae1"}
Nov 02 14:39:33.243312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:33.243258    4492 scope.go:122] "RemoveContainer" containerID="91397fe69ff0eb20433d2ffebc5068c9ef9d7d90850c9aadceb56a66e1b689e3"
Nov 02 14:39:33.298919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:33.298883    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"]
Nov 02 14:39:33.317335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:33.317299    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-555d1e-6d9867c5f9-drnpk"]
Nov 02 14:39:33.581611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:33.581525    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" path="/var/lib/kubelet/pods/b907231f-8cb0-4bed-86e5-898b4145d948/volumes"
Nov 02 14:39:34.396400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:34.396349    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:34.690718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:34.690670    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:35.497140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.497113    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:39:35.550206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550175    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550237    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550276    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550307    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550334    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550379    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550448    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550482    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550508    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9\") pod \"677cda29-e1cc-4b89-b851-9460141412a7\" (UID: \"677cda29-e1cc-4b89-b851-9460141412a7\") "
Nov 02 14:39:35.550718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550529    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:35.550718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.550712    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:35.551180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.551079    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:39:35.551180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.551102    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.551180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.551116    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:35.551180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.551123    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.551180 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.551147    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:35.553041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.553013    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:39:35.553145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.553041    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:35.553518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.553500    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:35.553607 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.553585    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9" pod "677cda29-e1cc-4b89-b851-9460141412a7" (UID: "677cda29-e1cc-4b89-b851-9460141412a7"). InnerVolumeSpecName "kube-api-access-rwks9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:35.651993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.651948    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.651993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.651982    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.651993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.651996    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/677cda29-e1cc-4b89-b851-9460141412a7-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.652309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.652009    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.652309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.652021    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/677cda29-e1cc-4b89-b851-9460141412a7-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.652309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.652033    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/677cda29-e1cc-4b89-b851-9460141412a7-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.652309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.652043    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwks9\" (UniqueName: \"kubernetes.io/projected/677cda29-e1cc-4b89-b851-9460141412a7-kube-api-access-rwks9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:35.948738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:35.948698    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:36.269760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.269671    4492 generic.go:356] "Generic (PLEG): container finished" podID="677cda29-e1cc-4b89-b851-9460141412a7" containerID="ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878" exitCode=0
Nov 02 14:39:36.269760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.269727    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"
Nov 02 14:39:36.269993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.269753    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" event={"ID":"677cda29-e1cc-4b89-b851-9460141412a7","Type":"ContainerDied","Data":"ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878"}
Nov 02 14:39:36.269993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.269825    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46" event={"ID":"677cda29-e1cc-4b89-b851-9460141412a7","Type":"ContainerDied","Data":"879469389d50ce09c290a1f1a77fbecf9dbb39eba984bf205825898e0c286f18"}
Nov 02 14:39:36.269993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.269866    4492 scope.go:122] "RemoveContainer" containerID="ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878"
Nov 02 14:39:36.280680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.280651    4492 scope.go:122] "RemoveContainer" containerID="d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"
Nov 02 14:39:36.291298 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.291272    4492 scope.go:122] "RemoveContainer" containerID="ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878"
Nov 02 14:39:36.291701 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:39:36.291680    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878\": container with ID starting with ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878 not found: ID does not exist" containerID="ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878"
Nov 02 14:39:36.291802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.291712    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878"} err="failed to get container status \"ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878\": rpc error: code = NotFound desc = could not find container \"ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878\": container with ID starting with ee9cc4b5cf696d4f897438446520f37b343717a1026fae8a6cc2a3b66258b878 not found: ID does not exist"
Nov 02 14:39:36.291802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.291733    4492 scope.go:122] "RemoveContainer" containerID="d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"
Nov 02 14:39:36.292032 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:39:36.292011    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4\": container with ID starting with d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4 not found: ID does not exist" containerID="d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"
Nov 02 14:39:36.292065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.292043    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4"} err="failed to get container status \"d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4\": rpc error: code = NotFound desc = could not find container \"d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4\": container with ID starting with d6134c3fd03acb9705d059e382b5b18394d2ddf1b7e5720618a801fbae733df4 not found: ID does not exist"
Nov 02 14:39:36.330596 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.330563    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"]
Nov 02 14:39:36.342709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.342654    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-2dt46"]
Nov 02 14:39:36.561691 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:36.561615    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:37.187674 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.187598    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:37.191066 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.191042    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:39:37.273780 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.273739    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.273997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.273792    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.273997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.273862    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.273997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.273925    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.274186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274036    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.274186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274145    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:37.274300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274168    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.274300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274183    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274450    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274574    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274646    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.274749    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token\") pod \"f225768f-8c3d-4184-9cdd-14de3500fbb6\" (UID: \"f225768f-8c3d-4184-9cdd-14de3500fbb6\") "
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.275246    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.275271    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.275290    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.275512    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:37.276399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.275666    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:39:37.277320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.277275    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:39:37.277320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.277284    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:39:37.277500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.277318    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "kube-api-access-6hmwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:37.277675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.277653    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token" pod "f225768f-8c3d-4184-9cdd-14de3500fbb6" (UID: "f225768f-8c3d-4184-9cdd-14de3500fbb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:39:37.280483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.280455    4492 generic.go:356] "Generic (PLEG): container finished" podID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerID="4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78" exitCode=0
Nov 02 14:39:37.280579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.280514    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"
Nov 02 14:39:37.280579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.280538    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" event={"ID":"f225768f-8c3d-4184-9cdd-14de3500fbb6","Type":"ContainerDied","Data":"4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78"}
Nov 02 14:39:37.280692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.280601    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm" event={"ID":"f225768f-8c3d-4184-9cdd-14de3500fbb6","Type":"ContainerDied","Data":"03386391fa783744b7885c445f9e0bf25ce2fd2147c2e94a786ac5be1790fa61"}
Nov 02 14:39:37.280692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.280621    4492 scope.go:122] "RemoveContainer" containerID="4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78"
Nov 02 14:39:37.297419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.297394    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:37.302988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.302954    4492 scope.go:122] "RemoveContainer" containerID="4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78"
Nov 02 14:39:37.303293 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:39:37.303274    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78\": container with ID starting with 4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78 not found: ID does not exist" containerID="4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78"
Nov 02 14:39:37.303452 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.303304    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78"} err="failed to get container status \"4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78\": rpc error: code = NotFound desc = could not find container \"4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78\": container with ID starting with 4d641c3d5912f36871943ddae318fb21bfe3a5ed1059d72651b8587256c73c78 not found: ID does not exist"
Nov 02 14:39:37.334088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.334057    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"]
Nov 02 14:39:37.344948 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.344909    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-wfqsm"]
Nov 02 14:39:37.376081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.376049    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmwt\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-kube-api-access-6hmwt\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.376081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.376075    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.376081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.376086    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f225768f-8c3d-4184-9cdd-14de3500fbb6-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.376302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.376095    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f225768f-8c3d-4184-9cdd-14de3500fbb6-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.376302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.376103    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.376302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.376111    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f225768f-8c3d-4184-9cdd-14de3500fbb6-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:39:37.581137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.581066    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677cda29-e1cc-4b89-b851-9460141412a7" path="/var/lib/kubelet/pods/677cda29-e1cc-4b89-b851-9460141412a7/volumes"
Nov 02 14:39:37.581532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:37.581511    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" path="/var/lib/kubelet/pods/f225768f-8c3d-4184-9cdd-14de3500fbb6/volumes"
Nov 02 14:39:38.358930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:38.358899    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:38.485220 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:38.485187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:40.075221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:40.075185    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:40.345309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:40.345225    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:40.996231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:40.996194    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:42.138147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:42.138109    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:43.934573 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:43.934539    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:45.745975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:45.745940    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:46.518081 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:39:46.518043    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:39:47.658779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:47.658733    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:48.741646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:48.741606    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"]
Nov 02 14:39:48.768694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:48.768655    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"]
Nov 02 14:39:48.880281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:48.880216    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:48.999455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:48.993500    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:49.239695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:49.239650    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:49.872394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:49.872338    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:50.070132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:50.070068    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:51.756042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:51.756011    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:53.582079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:53.582030    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:53.769294 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:53.769250    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerName="payload-pre-processing" containerID="cri-o://31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6" gracePeriod=25
Nov 02 14:39:54.379831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:54.379750    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerName="payload-processing" containerID="cri-o://102ad6ce4cfc5acd5daf63f24b22a19dc6fb69829b147d828226ac8806591e8b" gracePeriod=25
Nov 02 14:39:55.355492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:55.355455    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:56.113058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:56.113025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:57.163938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:57.163864    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:58.924771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:58.924722    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:59.725058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:59.725023    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:39:59.845552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:39:59.845522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:00.827388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:00.827327    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:02.625929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:02.625895    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:03.316308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:03.316267    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:40:03.316308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:03.316285    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:40:04.239958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:04.239919    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:04.426903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:04.426865    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:06.256244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:06.256210    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:07.013400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:07.013340    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:08.118071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:08.116943    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:09.949437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:09.949398    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:10.100544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:10.100494    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:10.698753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:10.698716    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:11.034699 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:11.034617    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:11.622522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:11.622487    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:12.489950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:12.489911    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:12.698346 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:12.698308    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:13.316267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:13.316235    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:40:13.316482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:13.316244    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:40:14.049074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.049052    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:40:14.169023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.168981    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume\") pod \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") "
Nov 02 14:40:14.169317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.169140    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm\" (UniqueName: \"kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm\") pod \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\" (UID: \"7b857731-b51d-4bcf-bdc6-69cfb034f2c5\") "
Nov 02 14:40:14.169470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.169444    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume" pod "7b857731-b51d-4bcf-bdc6-69cfb034f2c5" (UID: "7b857731-b51d-4bcf-bdc6-69cfb034f2c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:14.171615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.171589    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm" pod "7b857731-b51d-4bcf-bdc6-69cfb034f2c5" (UID: "7b857731-b51d-4bcf-bdc6-69cfb034f2c5"). InnerVolumeSpecName "kube-api-access-dr9wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:14.246897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.246862    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:14.270335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.270295    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9wm\" (UniqueName: \"kubernetes.io/projected/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-kube-api-access-dr9wm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:40:14.270335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.270326    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b857731-b51d-4bcf-bdc6-69cfb034f2c5-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:40:14.572013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.571976    4492 generic.go:356] "Generic (PLEG): container finished" podID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerID="102ad6ce4cfc5acd5daf63f24b22a19dc6fb69829b147d828226ac8806591e8b" exitCode=0
Nov 02 14:40:14.572192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.572044    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" event={"ID":"262b06a7-8588-40d3-8da0-8f8b643b5fae","Type":"ContainerDied","Data":"102ad6ce4cfc5acd5daf63f24b22a19dc6fb69829b147d828226ac8806591e8b"}
Nov 02 14:40:14.574945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.574904    4492 generic.go:356] "Generic (PLEG): container finished" podID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerID="31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6" exitCode=0
Nov 02 14:40:14.574945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.574941    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" event={"ID":"7b857731-b51d-4bcf-bdc6-69cfb034f2c5","Type":"ContainerDied","Data":"31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6"}
Nov 02 14:40:14.575160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.574949    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"
Nov 02 14:40:14.575160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.574999    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm" event={"ID":"7b857731-b51d-4bcf-bdc6-69cfb034f2c5","Type":"ContainerDied","Data":"cd2662eac0436b79214a039a82dfe1695e69dfacaa35154d6367e5d236378802"}
Nov 02 14:40:14.575160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.575023    4492 scope.go:122] "RemoveContainer" containerID="31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6"
Nov 02 14:40:14.615636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.615613    4492 scope.go:122] "RemoveContainer" containerID="31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6"
Nov 02 14:40:14.616593 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:40:14.616533    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6\": container with ID starting with 31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6 not found: ID does not exist" containerID="31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6"
Nov 02 14:40:14.616721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.616608    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6"} err="failed to get container status \"31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6\": rpc error: code = NotFound desc = could not find container \"31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6\": container with ID starting with 31797ff4e94216cd59659ec47e55b1f938d2bf048e03f2b7b95433f38c2b43b6 not found: ID does not exist"
Nov 02 14:40:14.639465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.639386    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:40:14.639465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.639437    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"]
Nov 02 14:40:14.642958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.642936    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c0c28f-68fd74bd7f-88snm"]
Nov 02 14:40:14.775191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.775162    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4\" (UniqueName: \"kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4\") pod \"262b06a7-8588-40d3-8da0-8f8b643b5fae\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") "
Nov 02 14:40:14.775440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.775302    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume\") pod \"262b06a7-8588-40d3-8da0-8f8b643b5fae\" (UID: \"262b06a7-8588-40d3-8da0-8f8b643b5fae\") "
Nov 02 14:40:14.775917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.775888    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume" pod "262b06a7-8588-40d3-8da0-8f8b643b5fae" (UID: "262b06a7-8588-40d3-8da0-8f8b643b5fae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:14.777564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.777536    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4" pod "262b06a7-8588-40d3-8da0-8f8b643b5fae" (UID: "262b06a7-8588-40d3-8da0-8f8b643b5fae"). InnerVolumeSpecName "kube-api-access-9thq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:14.876817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.876781    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thq4\" (UniqueName: \"kubernetes.io/projected/262b06a7-8588-40d3-8da0-8f8b643b5fae-kube-api-access-9thq4\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:40:14.876817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:14.876810    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262b06a7-8588-40d3-8da0-8f8b643b5fae-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:40:15.583648 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:15.583617    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" path="/var/lib/kubelet/pods/7b857731-b51d-4bcf-bdc6-69cfb034f2c5/volumes"
Nov 02 14:40:15.587134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:15.587110    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"
Nov 02 14:40:15.587275 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:15.587138    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5" event={"ID":"262b06a7-8588-40d3-8da0-8f8b643b5fae","Type":"ContainerDied","Data":"0f36cccc6f9cdc68511d1be1546eb661c3b8c46f8e4a80183c9ff51bf8a0fe68"}
Nov 02 14:40:15.587275 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:15.587195    4492 scope.go:122] "RemoveContainer" containerID="102ad6ce4cfc5acd5daf63f24b22a19dc6fb69829b147d828226ac8806591e8b"
Nov 02 14:40:15.658814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:15.658761    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"]
Nov 02 14:40:15.692145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:15.692105    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c0c28f-7854685bb7-dqdp5"]
Nov 02 14:40:16.143654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:16.143608    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:16.258321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:16.258272    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:16.517201 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:40:16.517158    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:40:17.582198 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:17.582123    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" path="/var/lib/kubelet/pods/262b06a7-8588-40d3-8da0-8f8b643b5fae/volumes"
Nov 02 14:40:18.035230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:18.035194    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:19.774444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:19.774409    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:19.993735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:19.993685    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:21.274061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:21.274026    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:21.396307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:21.396274    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:21.786690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:21.786654    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:22.072341 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:22.072257    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:22.536710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:22.536674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:23.832558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:23.832507    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:24.579831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:24.579649    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:24.702614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:24.702585    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:25.765185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:25.765146    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:27.628773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:27.628741    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:29.436102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:29.436055    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:30.854601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.854569    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"]
Nov 02 14:40:30.855136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855123    4492 cpu_manager.go:414] "containerMap: removing container" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy"
Nov 02 14:40:30.855184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855137    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy"
Nov 02 14:40:30.855184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855153    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy"
Nov 02 14:40:30.855184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855160    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b90407-cb9e-400b-a369-e04b59cde7cc" containerName="istio-proxy"
Nov 02 14:40:30.855184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855172    4492 cpu_manager.go:414] "containerMap: removing container" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy"
Nov 02 14:40:30.855184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855177    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy"
Nov 02 14:40:30.855184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855184    4492 cpu_manager.go:414] "containerMap: removing container" podUID="329d2121-eaad-4b40-af4c-31ec54cd398b" containerName="maas-api"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855188    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329d2121-eaad-4b40-af4c-31ec54cd398b" containerName="maas-api"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855197    4492 cpu_manager.go:414] "containerMap: removing container" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerName="payload-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855201    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerName="payload-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855208    4492 cpu_manager.go:414] "containerMap: removing container" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855212    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855224    4492 cpu_manager.go:414] "containerMap: removing container" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerName="payload-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855228    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerName="payload-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855237    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855241    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855250    4492 cpu_manager.go:414] "containerMap: removing container" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855254    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855262    4492 cpu_manager.go:414] "containerMap: removing container" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855266    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855271    4492 cpu_manager.go:414] "containerMap: removing container" podUID="de86f7d7-e096-429a-96d7-828c085393c9" containerName="maas-api"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855275    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de86f7d7-e096-429a-96d7-828c085393c9" containerName="maas-api"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855283    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855287    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855296    4492 cpu_manager.go:414] "containerMap: removing container" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855300    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7177b655-6676-4a9f-8a9a-b9d4ab7f59c3" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855392    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0ef4bd-8b1c-431f-a050-6e6d7ae2d93b" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855403    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3d5ac5-3b5d-4e4f-917c-e44c137c453d" containerName="istio-proxy"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855411    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b857731-b51d-4bcf-bdc6-69cfb034f2c5" containerName="payload-pre-processing"
Nov 02 14:40:30.855413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855418    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="de86f7d7-e096-429a-96d7-828c085393c9" containerName="maas-api"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855425    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="b907231f-8cb0-4bed-86e5-898b4145d948" containerName="payload-pre-processing"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855432    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855441    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="f225768f-8c3d-4184-9cdd-14de3500fbb6" containerName="istio-proxy"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855449    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="329d2121-eaad-4b40-af4c-31ec54cd398b" containerName="maas-api"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855458    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="262b06a7-8588-40d3-8da0-8f8b643b5fae" containerName="payload-processing"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855463    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="677cda29-e1cc-4b89-b851-9460141412a7" containerName="istio-proxy"
Nov 02 14:40:30.856093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.855468    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="7181ff04-9a1e-40c8-b261-81ffcc373eef" containerName="payload-processing"
Nov 02 14:40:30.860727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.860707    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.874803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.874776    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.874923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.874815    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.874923 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.874889    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.875045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.874947    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.875045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.874964    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.875186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.875161    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.875235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.875209    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.875393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.875376    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.875433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.875407    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdxs\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.889624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.889592    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"]
Nov 02 14:40:30.962530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.962494    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-866159-openshift-default-dockercfg-jpxc8\""
Nov 02 14:40:30.976394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976346    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976407    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976514    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976542    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdxs\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976600    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976635    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976674    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976752    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976779    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.976893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976865    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.977042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.976930    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.977098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.977044    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.977158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.977105    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.977370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.977334    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.978958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.978936    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.979071 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.979026    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.998346 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.998314    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:30.998733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:30.998713    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdxs\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:31.173903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.173857    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:31.176819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.176801    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:31.528129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.528102    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"]
Nov 02 14:40:31.531485 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:40:31.531458    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0912ef0_0fcb_4dfb_8c36_73e809947187.slice/crio-cbfd0ac30fc04c678692de7dd2a836fd83caba0a4651574da0b08cf1186c2b6e WatchSource:0}: Error finding container cbfd0ac30fc04c678692de7dd2a836fd83caba0a4651574da0b08cf1186c2b6e: Status 404 returned error can't find the container with id cbfd0ac30fc04c678692de7dd2a836fd83caba0a4651574da0b08cf1186c2b6e
Nov 02 14:40:31.533821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.533790    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:40:31.533916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.533864    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:40:31.533958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.533913    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:40:31.758690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.758657    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" event={"ID":"b0912ef0-0fcb-4dfb-8c36-73e809947187","Type":"ContainerStarted","Data":"d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73"}
Nov 02 14:40:31.758876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.758718    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" event={"ID":"b0912ef0-0fcb-4dfb-8c36-73e809947187","Type":"ContainerStarted","Data":"cbfd0ac30fc04c678692de7dd2a836fd83caba0a4651574da0b08cf1186c2b6e"}
Nov 02 14:40:31.801938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:31.801763    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podStartSLOduration=1.801746965 podStartE2EDuration="1.801746965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:31.800336623 +0000 UTC m=+2596.344052221" watchObservedRunningTime="2026-11-02 14:40:31.801746965 +0000 UTC m=+2596.345462559"
Nov 02 14:40:32.177565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:32.177539    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:32.179532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:32.179506    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:32.179656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:32.179553    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:32.260307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:32.259021    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:32.896134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:32.896103    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:33.024182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:33.024148    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:33.177332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:33.177303    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:33.177500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:33.177385    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:34.177578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:34.177539    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:34.178019 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:34.177600    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:34.594706 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:34.594618    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:34.758377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:34.750896    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:35.177261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:35.177224    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:35.177482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:35.177292    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:35.283091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:35.283051    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:36.177895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:36.177826    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:36.178111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:36.177960    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:36.648081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:36.647958    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:37.177425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:37.177379    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:37.177648 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:37.177443    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:38.178224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:38.178181    4492 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:40:38.178677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:38.178255    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 14:40:38.337756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:38.337714    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:38.952345 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:38.952309    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:39.239928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:39.239842    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:39.240669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:39.240645    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:39.241779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:39.241710    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:40:40.193823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:40.193780    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:40.371892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:40.371857    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:41.720258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:41.720210    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"]
Nov 02 14:40:41.747409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:41.747378    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:41.874715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:41.874678    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:41.986739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:41.986663    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:42.564302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:42.564270    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:43.711332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:43.711305    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:44.510213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:44.509631    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:46.327009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:46.326965    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:46.517512 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:40:46.517473    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:40:46.720835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:46.720795    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerName="payload-processing" containerID="cri-o://826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d" gracePeriod=25
Nov 02 14:40:48.116501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:48.116456    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:49.744021 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:49.743981    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:49.858064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:49.858031    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:50.243533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:50.243499    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:40:50.501615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:50.501522    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:50.936866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:50.936825    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:51.725026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:51.724979    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:51.831754 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:51.831723    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:53.526756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:53.526708    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:55.347784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:55.347751    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:57.130826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:57.130783    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:57.444194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:57.444164    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:57.554809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:57.554779    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:40:59.023259 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:40:59.023226    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:00.049463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.044822    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:00.132950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.132911    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"]
Nov 02 14:41:00.243826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.243788    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:41:00.267586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.267534    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:00.385538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.385437    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:00.613806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.613774    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"]
Nov 02 14:41:00.614045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.614020    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" podUID="bfe72977-06e8-46e5-9c35-cb99090fd84a" containerName="maas-api" containerID="cri-o://36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3" gracePeriod=30
Nov 02 14:41:00.674997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.674956    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:00.960074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.960038    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:41:00.985906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.985848    4492 generic.go:356] "Generic (PLEG): container finished" podID="bfe72977-06e8-46e5-9c35-cb99090fd84a" containerID="36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3" exitCode=0
Nov 02 14:41:00.986102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.985913    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"
Nov 02 14:41:00.986102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.985933    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" event={"ID":"bfe72977-06e8-46e5-9c35-cb99090fd84a","Type":"ContainerDied","Data":"36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3"}
Nov 02 14:41:00.986102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.985986    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d" event={"ID":"bfe72977-06e8-46e5-9c35-cb99090fd84a","Type":"ContainerDied","Data":"7b1d24a5fb9eaad791442499a7c08645e3eaf2f8da5d9fbabd3b3f777dbac787"}
Nov 02 14:41:00.986102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.986012    4492 scope.go:122] "RemoveContainer" containerID="36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3"
Nov 02 14:41:00.995442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.995424    4492 scope.go:122] "RemoveContainer" containerID="36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3"
Nov 02 14:41:00.995704 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:00.995687    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3\": container with ID starting with 36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3 not found: ID does not exist" containerID="36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3"
Nov 02 14:41:00.995765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:00.995715    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3"} err="failed to get container status \"36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3\": rpc error: code = NotFound desc = could not find container \"36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3\": container with ID starting with 36e10ef24fec7c14b0a42e96c97f5f2bcf21e08e771fbc14c2a7203b340794d3 not found: ID does not exist"
Nov 02 14:41:01.077741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.077698    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs\") pod \"bfe72977-06e8-46e5-9c35-cb99090fd84a\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") "
Nov 02 14:41:01.077741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.077753    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf\" (UniqueName: \"kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf\") pod \"bfe72977-06e8-46e5-9c35-cb99090fd84a\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") "
Nov 02 14:41:01.078348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.078032    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca\") pod \"bfe72977-06e8-46e5-9c35-cb99090fd84a\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") "
Nov 02 14:41:01.078348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.078104    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\") pod \"bfe72977-06e8-46e5-9c35-cb99090fd84a\" (UID: \"bfe72977-06e8-46e5-9c35-cb99090fd84a\") "
Nov 02 14:41:01.082913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.082861    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca" pod "bfe72977-06e8-46e5-9c35-cb99090fd84a" (UID: "bfe72977-06e8-46e5-9c35-cb99090fd84a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:41:01.102438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.102383    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf" pod "bfe72977-06e8-46e5-9c35-cb99090fd84a" (UID: "bfe72977-06e8-46e5-9c35-cb99090fd84a"). InnerVolumeSpecName "kube-api-access-ffdmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:41:01.102746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.102708    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs" pod "bfe72977-06e8-46e5-9c35-cb99090fd84a" (UID: "bfe72977-06e8-46e5-9c35-cb99090fd84a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:41:01.104815 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.104783    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls" pod "bfe72977-06e8-46e5-9c35-cb99090fd84a" (UID: "bfe72977-06e8-46e5-9c35-cb99090fd84a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:41:01.179901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.179810    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72977-06e8-46e5-9c35-cb99090fd84a-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:01.179901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.179847    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:01.179901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.179865    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe72977-06e8-46e5-9c35-cb99090fd84a-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:01.179901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.179879    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffdmf\" (UniqueName: \"kubernetes.io/projected/bfe72977-06e8-46e5-9c35-cb99090fd84a-kube-api-access-ffdmf\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:01.331277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.331246    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"]
Nov 02 14:41:01.337804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.337777    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5ba53f-fc5859669-2gh7d"]
Nov 02 14:41:01.390178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.390153    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:01.580594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:01.580506    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe72977-06e8-46e5-9c35-cb99090fd84a" path="/var/lib/kubelet/pods/bfe72977-06e8-46e5-9c35-cb99090fd84a/volumes"
Nov 02 14:41:03.130325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:03.130291    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:04.921162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:04.921129    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:05.133319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:05.133268    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerName="payload-processing" containerID="cri-o://e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae" gracePeriod=25
Nov 02 14:41:06.737057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:06.736991    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:06.991204 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:06.991131    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:41:07.025928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.025885    4492 generic.go:356] "Generic (PLEG): container finished" podID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerID="826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d" exitCode=0
Nov 02 14:41:07.026122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.025945    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"
Nov 02 14:41:07.026122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.025955    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" event={"ID":"3e528bec-5bc1-4638-9549-d6bbbdc486af","Type":"ContainerDied","Data":"826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d"}
Nov 02 14:41:07.026122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.025999    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd" event={"ID":"3e528bec-5bc1-4638-9549-d6bbbdc486af","Type":"ContainerDied","Data":"a1f4960a96e6ddaa8b977dba920b340922d34757e1735e4a7355cc109409ed5e"}
Nov 02 14:41:07.026122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.026027    4492 scope.go:122] "RemoveContainer" containerID="826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d"
Nov 02 14:41:07.037559 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.037533    4492 scope.go:122] "RemoveContainer" containerID="826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d"
Nov 02 14:41:07.037922 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:07.037899    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d\": container with ID starting with 826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d not found: ID does not exist" containerID="826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d"
Nov 02 14:41:07.038006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.037933    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d"} err="failed to get container status \"826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d\": rpc error: code = NotFound desc = could not find container \"826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d\": container with ID starting with 826879fab02c28978693376c192f722df9a57d202462bd78ce59df9be16f156d not found: ID does not exist"
Nov 02 14:41:07.153421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.153389    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l\" (UniqueName: \"kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l\") pod \"3e528bec-5bc1-4638-9549-d6bbbdc486af\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") "
Nov 02 14:41:07.153687 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.153663    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume\") pod \"3e528bec-5bc1-4638-9549-d6bbbdc486af\" (UID: \"3e528bec-5bc1-4638-9549-d6bbbdc486af\") "
Nov 02 14:41:07.154436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.154386    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume" pod "3e528bec-5bc1-4638-9549-d6bbbdc486af" (UID: "3e528bec-5bc1-4638-9549-d6bbbdc486af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:41:07.155055 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.155032    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e528bec-5bc1-4638-9549-d6bbbdc486af-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:07.156863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.156837    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l" pod "3e528bec-5bc1-4638-9549-d6bbbdc486af" (UID: "3e528bec-5bc1-4638-9549-d6bbbdc486af"). InnerVolumeSpecName "kube-api-access-26f4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:41:07.256441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.256345    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26f4l\" (UniqueName: \"kubernetes.io/projected/3e528bec-5bc1-4638-9549-d6bbbdc486af-kube-api-access-26f4l\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:07.375400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.375339    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"]
Nov 02 14:41:07.383704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.383672    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e07583-7749bbd66c-mkjfd"]
Nov 02 14:41:07.580920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.580836    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" path="/var/lib/kubelet/pods/3e528bec-5bc1-4638-9549-d6bbbdc486af/volumes"
Nov 02 14:41:07.600421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:07.600392    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:08.531534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:08.531499    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:10.196805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:10.196771    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:10.330067 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:10.330032    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:10.957043 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:10.957009    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:41:12.051287 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:12.051251    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:14.245291 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:14.245258    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:14.682152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:14.682121    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:15.990326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:15.990284    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:16.517197 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:16.517164    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:41:17.911528 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:17.911494    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:19.606444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:19.606388    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:19.716840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:19.716799    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:20.168424 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:20.168387    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:20.957111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:20.957077    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:41:21.503016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:21.502972    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:23.211908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:23.211874    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:23.986788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:23.986752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:24.101544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:24.101510    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:24.747524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:24.747495    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:25.022669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.022588    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:25.155313 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.155280    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:25.478052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.478026    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:41:25.585547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.585469    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume\") pod \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") "
Nov 02 14:41:25.585705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.585614    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6\" (UniqueName: \"kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6\") pod \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\" (UID: \"e5c683bc-3642-4ce9-84e5-cb88f26e87b4\") "
Nov 02 14:41:25.585930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.585900    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume" pod "e5c683bc-3642-4ce9-84e5-cb88f26e87b4" (UID: "e5c683bc-3642-4ce9-84e5-cb88f26e87b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:41:25.586064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.586032    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:25.587950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.587920    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6" pod "e5c683bc-3642-4ce9-84e5-cb88f26e87b4" (UID: "e5c683bc-3642-4ce9-84e5-cb88f26e87b4"). InnerVolumeSpecName "kube-api-access-jvjj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:41:25.687312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:25.687278    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjj6\" (UniqueName: \"kubernetes.io/projected/e5c683bc-3642-4ce9-84e5-cb88f26e87b4-kube-api-access-jvjj6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:26.011257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.011224    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:26.176847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.176658    4492 generic.go:356] "Generic (PLEG): container finished" podID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerID="e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae" exitCode=0
Nov 02 14:41:26.176847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.176777    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" event={"ID":"e5c683bc-3642-4ce9-84e5-cb88f26e87b4","Type":"ContainerDied","Data":"e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae"}
Nov 02 14:41:26.176847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.176837    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"
Nov 02 14:41:26.177140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.176887    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784" event={"ID":"e5c683bc-3642-4ce9-84e5-cb88f26e87b4","Type":"ContainerDied","Data":"acfcbb532a23568fd13de419623baabdda1d4288c84570bc4e13470f1ffb0414"}
Nov 02 14:41:26.177140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.176919    4492 scope.go:122] "RemoveContainer" containerID="e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae"
Nov 02 14:41:26.187311 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.187293    4492 scope.go:122] "RemoveContainer" containerID="e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae"
Nov 02 14:41:26.187619 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:26.187596    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae\": container with ID starting with e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae not found: ID does not exist" containerID="e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae"
Nov 02 14:41:26.187698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.187631    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae"} err="failed to get container status \"e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae\": rpc error: code = NotFound desc = could not find container \"e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae\": container with ID starting with e659a8e12e3130a7f9618f586e40d303119cab55cbbc7357d91576a2ecb06aae not found: ID does not exist"
Nov 02 14:41:26.226319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.226280    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"]
Nov 02 14:41:26.236140 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.236111    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ba53f-85689465fc-jq784"]
Nov 02 14:41:26.815769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:26.815730    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:27.581908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:27.581860    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" path="/var/lib/kubelet/pods/e5c683bc-3642-4ce9-84e5-cb88f26e87b4/volumes"
Nov 02 14:41:28.505511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:28.505476    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:30.256886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:30.256853    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:31.853776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:31.853737    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:31.976796 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:31.971776    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:32.168317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:32.168238    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:33.367197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:33.367165    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:33.920093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:33.920061    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:35.714365 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:35.714330    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:37.204843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:37.204807    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:37.846982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:37.846933    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:39.601665 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:39.601632    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:39.762284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:39.762242    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:39.879223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:39.879130    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:40.344588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:40.344553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:40.555538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:40.555504    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:40.955051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:40.955015    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:41.636846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:41.636812    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:41.765583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:41.765548    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:43.534851 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:43.534818    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:45.347442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:45.347404    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:46.518142 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:46.518103    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:41:47.143324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:47.143294    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:48.736767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:48.736729    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"]
Nov 02 14:41:48.737255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:48.737019    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f" podUID="8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" containerName="maas-api" containerID="cri-o://a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26" gracePeriod=30
Nov 02 14:41:48.799916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:48.799874    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:48.925347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:48.925238    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:48.949298 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:48.949260    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"]
Nov 02 14:41:48.972834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:48.972805    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"]
Nov 02 14:41:49.007275 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.007204    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:41:49.052624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.052593    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:49.101296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.101263    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs\") pod \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") "
Nov 02 14:41:49.101524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.101414    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca\") pod \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") "
Nov 02 14:41:49.101524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.101457    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx\" (UniqueName: \"kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx\") pod \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") "
Nov 02 14:41:49.101524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.101512    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls\") pod \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\" (UID: \"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4\") "
Nov 02 14:41:49.101841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.101798    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca" pod "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" (UID: "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:41:49.102119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.102096    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:49.104095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.104072    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls" pod "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" (UID: "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:41:49.104247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.104225    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx" pod "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" (UID: "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4"). InnerVolumeSpecName "kube-api-access-dnxlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:41:49.104295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.104276    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs" pod "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" (UID: "8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:41:49.164532 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.164495    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:49.203184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.203153    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:49.203184 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.203186    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnxlx\" (UniqueName: \"kubernetes.io/projected/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-kube-api-access-dnxlx\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:49.203444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.203200    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:49.350427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.344486    4492 generic.go:356] "Generic (PLEG): container finished" podID="8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" containerID="a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26" exitCode=0
Nov 02 14:41:49.350427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.344568    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f" event={"ID":"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4","Type":"ContainerDied","Data":"a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26"}
Nov 02 14:41:49.350427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.344623    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f" event={"ID":"8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4","Type":"ContainerDied","Data":"8c665ffcffeba1ad4fa701b9fa86bfd9c403c5e79d2eb33c925411f0216c87ad"}
Nov 02 14:41:49.350427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.344653    4492 scope.go:122] "RemoveContainer" containerID="a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26"
Nov 02 14:41:49.350427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.344837    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"
Nov 02 14:41:49.375564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.375528    4492 scope.go:122] "RemoveContainer" containerID="a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26"
Nov 02 14:41:49.376029 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:49.376006    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26\": container with ID starting with a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26 not found: ID does not exist" containerID="a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26"
Nov 02 14:41:49.376127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.376039    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26"} err="failed to get container status \"a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26\": rpc error: code = NotFound desc = could not find container \"a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26\": container with ID starting with a75113d176396496f00be13cafaf781a3917f25327c19e4733c8ef6e0db95b26 not found: ID does not exist"
Nov 02 14:41:49.413348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.413309    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"]
Nov 02 14:41:49.417325 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.417298    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-673ba8-7644bb54d9-54f8f"]
Nov 02 14:41:49.583513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.583470    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" path="/var/lib/kubelet/pods/8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4/volumes"
Nov 02 14:41:49.624649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:49.624567    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:51.628152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:51.628117    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:53.436982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:53.436949    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:53.950286 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:53.950238    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb" podUID="ab230828-8405-43cf-82c9-ff18683d733e" containerName="payload-processing" containerID="cri-o://7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8" gracePeriod=25
Nov 02 14:41:53.974074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:53.974035    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk" podUID="e1d9f9ec-f170-4d5e-973c-7e03510570ab" containerName="payload-pre-processing" containerID="cri-o://288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78" gracePeriod=25
Nov 02 14:41:54.252114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.252086    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:41:54.296472 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.296445    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:41:54.369839 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.369778    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume\") pod \"ab230828-8405-43cf-82c9-ff18683d733e\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") "
Nov 02 14:41:54.370075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.369859    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g\" (UniqueName: \"kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g\") pod \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") "
Nov 02 14:41:54.370075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.369963    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r\" (UniqueName: \"kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r\") pod \"ab230828-8405-43cf-82c9-ff18683d733e\" (UID: \"ab230828-8405-43cf-82c9-ff18683d733e\") "
Nov 02 14:41:54.370075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.370070    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume\") pod \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\" (UID: \"e1d9f9ec-f170-4d5e-973c-7e03510570ab\") "
Nov 02 14:41:54.370319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.370290    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume" pod "ab230828-8405-43cf-82c9-ff18683d733e" (UID: "ab230828-8405-43cf-82c9-ff18683d733e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:41:54.370679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.370657    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab230828-8405-43cf-82c9-ff18683d733e-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:54.370840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.370715    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume" pod "e1d9f9ec-f170-4d5e-973c-7e03510570ab" (UID: "e1d9f9ec-f170-4d5e-973c-7e03510570ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:41:54.375548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.374876    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r" pod "ab230828-8405-43cf-82c9-ff18683d733e" (UID: "ab230828-8405-43cf-82c9-ff18683d733e"). InnerVolumeSpecName "kube-api-access-rhs9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:41:54.375548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.374985    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g" pod "e1d9f9ec-f170-4d5e-973c-7e03510570ab" (UID: "e1d9f9ec-f170-4d5e-973c-7e03510570ab"). InnerVolumeSpecName "kube-api-access-lx25g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:41:54.384855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.384819    4492 generic.go:356] "Generic (PLEG): container finished" podID="ab230828-8405-43cf-82c9-ff18683d733e" containerID="7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8" exitCode=0
Nov 02 14:41:54.385046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.384861    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"
Nov 02 14:41:54.385046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.384906    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb" event={"ID":"ab230828-8405-43cf-82c9-ff18683d733e","Type":"ContainerDied","Data":"7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8"}
Nov 02 14:41:54.385046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.384959    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb" event={"ID":"ab230828-8405-43cf-82c9-ff18683d733e","Type":"ContainerDied","Data":"608f1613572fccaa2287243c3cf3909b3a689f81c4bdeca071e2fbe5af5afdd5"}
Nov 02 14:41:54.385046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.384987    4492 scope.go:122] "RemoveContainer" containerID="7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8"
Nov 02 14:41:54.386246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.386225    4492 generic.go:356] "Generic (PLEG): container finished" podID="e1d9f9ec-f170-4d5e-973c-7e03510570ab" containerID="288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78" exitCode=0
Nov 02 14:41:54.386324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.386271    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"
Nov 02 14:41:54.386324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.386310    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk" event={"ID":"e1d9f9ec-f170-4d5e-973c-7e03510570ab","Type":"ContainerDied","Data":"288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78"}
Nov 02 14:41:54.386442 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.386377    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk" event={"ID":"e1d9f9ec-f170-4d5e-973c-7e03510570ab","Type":"ContainerDied","Data":"f6bf1e33322e4008a28ac1cea057968a94c8d9d8d930287c80e00448c1063346"}
Nov 02 14:41:54.395827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.395803    4492 scope.go:122] "RemoveContainer" containerID="7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8"
Nov 02 14:41:54.396160 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:54.396133    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8\": container with ID starting with 7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8 not found: ID does not exist" containerID="7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8"
Nov 02 14:41:54.396263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.396166    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8"} err="failed to get container status \"7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8\": rpc error: code = NotFound desc = could not find container \"7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8\": container with ID starting with 7734096e1d092d1e13bd1becdb60c6a296e1769f20971124e17748df2dcab5a8 not found: ID does not exist"
Nov 02 14:41:54.396263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.396184    4492 scope.go:122] "RemoveContainer" containerID="288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78"
Nov 02 14:41:54.405447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.405425    4492 scope.go:122] "RemoveContainer" containerID="288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78"
Nov 02 14:41:54.405758 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:41:54.405731    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78\": container with ID starting with 288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78 not found: ID does not exist" containerID="288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78"
Nov 02 14:41:54.405894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.405766    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78"} err="failed to get container status \"288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78\": rpc error: code = NotFound desc = could not find container \"288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78\": container with ID starting with 288b5aee160083884035e867ba02f655c32d2f9051181fc2815cd8d18ca1ec78 not found: ID does not exist"
Nov 02 14:41:54.433697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.433662    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"]
Nov 02 14:41:54.439249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.439216    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-673ba8-7664594f7f-zrsqb"]
Nov 02 14:41:54.469300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.469216    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"]
Nov 02 14:41:54.471587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.471559    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx25g\" (UniqueName: \"kubernetes.io/projected/e1d9f9ec-f170-4d5e-973c-7e03510570ab-kube-api-access-lx25g\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:54.471587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.471585    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhs9r\" (UniqueName: \"kubernetes.io/projected/ab230828-8405-43cf-82c9-ff18683d733e-kube-api-access-rhs9r\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:54.471817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.471595    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d9f9ec-f170-4d5e-973c-7e03510570ab-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:41:54.475911 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:54.475877    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-673ba8-6ffb686999-d8kvk"]
Nov 02 14:41:55.220650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:55.220569    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:55.382715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:55.382683    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:55.516686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:55.516598    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:55.581028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:55.580999    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab230828-8405-43cf-82c9-ff18683d733e" path="/var/lib/kubelet/pods/ab230828-8405-43cf-82c9-ff18683d733e/volumes"
Nov 02 14:41:55.581413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:55.581400    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d9f9ec-f170-4d5e-973c-7e03510570ab" path="/var/lib/kubelet/pods/e1d9f9ec-f170-4d5e-973c-7e03510570ab/volumes"
Nov 02 14:41:56.135385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:56.135334    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:56.528721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:56.528689    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:57.007051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:57.007016    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:57.195018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:57.194982    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:41:59.045946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:41:59.045906    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:00.923098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:00.923068    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:01.983723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:01.983690    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"]
Nov 02 14:42:01.996778 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:01.996737    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"]
Nov 02 14:42:02.091715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:02.091675    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:02.839223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:02.839187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:02.962404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:02.962335    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:03.073165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:03.073124    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:04.738987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:04.738935    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:06.535370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:06.535320    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:07.941853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:07.941809    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:08.588265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:08.588227    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:08.717742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:08.717694    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:08.817924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:08.817810    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"]
Nov 02 14:42:08.862170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:08.862139    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"]
Nov 02 14:42:08.974989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:08.974949    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:09.092300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:09.092212    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:10.324387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:10.324339    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:10.663920 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:10.663822    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:10.957669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:10.957611    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:12.540471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:12.540421    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:13.818330 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:13.818275    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerName="payload-processing" containerID="cri-o://f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917" gracePeriod=25
Nov 02 14:42:13.863252 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:13.863177    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerName="payload-pre-processing" containerID="cri-o://c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52" gracePeriod=25
Nov 02 14:42:14.315052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:14.315020    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"]
Nov 02 14:42:14.362079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:14.362050    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:14.473393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:14.473336    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:14.613296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:14.613212    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:15.184261 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:15.184227    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:42:15.897891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:15.897839    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:16.190758 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:16.190721    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:42:16.355059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:16.355028    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:16.519113 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:16.519029    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:42:17.005576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.005542    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" podUID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" containerName="main" containerID="cri-o://ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4" gracePeriod=45
Nov 02 14:42:17.017668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.017611    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" podUID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" containerName="main" containerID="cri-o://054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9" gracePeriod=45
Nov 02 14:42:17.150789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.150741    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" podUID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": dial tcp 10.134.0.93:8000: connect: connection refused"
Nov 02 14:42:17.195212 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.195084    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:17.301748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.301723    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:42:17.319444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.319421    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:42:17.410000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.409961    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache\") pod \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") "
Nov 02 14:42:17.410152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410031    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home\") pod \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") "
Nov 02 14:42:17.410152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410069    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs\") pod \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") "
Nov 02 14:42:17.410152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410129    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs\" (UniqueName: \"kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs\") pod \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") "
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410189    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh\" (UniqueName: \"kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh\") pod \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") "
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410232    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home\") pod \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") "
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410254    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache" pod "26d970fc-38ab-48a8-aab2-984e07c2b1d3" (UID: "26d970fc-38ab-48a8-aab2-984e07c2b1d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410270    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache\") pod \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") "
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410334    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir\") pod \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") "
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410335    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home" pod "26d970fc-38ab-48a8-aab2-984e07c2b1d3" (UID: "26d970fc-38ab-48a8-aab2-984e07c2b1d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.410378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410376    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm\") pod \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") "
Nov 02 14:42:17.410768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410413    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4\" (UniqueName: \"kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4\") pod \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") "
Nov 02 14:42:17.410768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410507    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir\") pod \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\" (UID: \"26d970fc-38ab-48a8-aab2-984e07c2b1d3\") "
Nov 02 14:42:17.410768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410528    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm\") pod \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\" (UID: \"fea0c720-e5b0-4bd1-b7ee-4946166fed00\") "
Nov 02 14:42:17.410768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410528    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache" pod "fea0c720-e5b0-4bd1-b7ee-4946166fed00" (UID: "fea0c720-e5b0-4bd1-b7ee-4946166fed00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.410768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410565    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir" pod "fea0c720-e5b0-4bd1-b7ee-4946166fed00" (UID: "fea0c720-e5b0-4bd1-b7ee-4946166fed00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.410768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410718    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home" pod "fea0c720-e5b0-4bd1-b7ee-4946166fed00" (UID: "fea0c720-e5b0-4bd1-b7ee-4946166fed00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.411113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410934    4492 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-model-cache\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.411113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410953    4492 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-home\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.411113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410965    4492 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-home\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.411113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.410989    4492 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-model-cache\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.411113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.411002    4492 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tmp-dir\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.411379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.411243    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir" pod "26d970fc-38ab-48a8-aab2-984e07c2b1d3" (UID: "26d970fc-38ab-48a8-aab2-984e07c2b1d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.413296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.413265    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs" pod "fea0c720-e5b0-4bd1-b7ee-4946166fed00" (UID: "fea0c720-e5b0-4bd1-b7ee-4946166fed00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:42:17.413543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.413331    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm" pod "fea0c720-e5b0-4bd1-b7ee-4946166fed00" (UID: "fea0c720-e5b0-4bd1-b7ee-4946166fed00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.413756 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.413738    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh" pod "fea0c720-e5b0-4bd1-b7ee-4946166fed00" (UID: "fea0c720-e5b0-4bd1-b7ee-4946166fed00"). InnerVolumeSpecName "kube-api-access-cdvkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:17.414210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.414055    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs" pod "26d970fc-38ab-48a8-aab2-984e07c2b1d3" (UID: "26d970fc-38ab-48a8-aab2-984e07c2b1d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:42:17.414210 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.414089    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm" pod "26d970fc-38ab-48a8-aab2-984e07c2b1d3" (UID: "26d970fc-38ab-48a8-aab2-984e07c2b1d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:17.419197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.419047    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4" pod "26d970fc-38ab-48a8-aab2-984e07c2b1d3" (UID: "26d970fc-38ab-48a8-aab2-984e07c2b1d3"). InnerVolumeSpecName "kube-api-access-rztv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511854    4492 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tls-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511880    4492 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea0c720-e5b0-4bd1-b7ee-4946166fed00-tls-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511890    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvkh\" (UniqueName: \"kubernetes.io/projected/fea0c720-e5b0-4bd1-b7ee-4946166fed00-kube-api-access-cdvkh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511899    4492 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-dshm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511907    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztv4\" (UniqueName: \"kubernetes.io/projected/26d970fc-38ab-48a8-aab2-984e07c2b1d3-kube-api-access-rztv4\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511916    4492 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d970fc-38ab-48a8-aab2-984e07c2b1d3-tmp-dir\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.511936 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.511923    4492 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea0c720-e5b0-4bd1-b7ee-4946166fed00-dshm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:17.551127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.551089    4492 generic.go:356] "Generic (PLEG): container finished" podID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" containerID="054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9" exitCode=0
Nov 02 14:42:17.551318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.551194    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"
Nov 02 14:42:17.551318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.551227    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" event={"ID":"26d970fc-38ab-48a8-aab2-984e07c2b1d3","Type":"ContainerDied","Data":"054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9"}
Nov 02 14:42:17.551318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.551276    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg" event={"ID":"26d970fc-38ab-48a8-aab2-984e07c2b1d3","Type":"ContainerDied","Data":"f694dbfeff2097b55251bda2501dff830685b1197e611c83b29d460f50bb3b9e"}
Nov 02 14:42:17.551318 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.551302    4492 scope.go:122] "RemoveContainer" containerID="054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9"
Nov 02 14:42:17.553997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.553971    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" event={"ID":"fea0c720-e5b0-4bd1-b7ee-4946166fed00","Type":"ContainerDied","Data":"ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4"}
Nov 02 14:42:17.554118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.553994    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"
Nov 02 14:42:17.554118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.553785    4492 generic.go:356] "Generic (PLEG): container finished" podID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" containerID="ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4" exitCode=0
Nov 02 14:42:17.554118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.554077    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b" event={"ID":"fea0c720-e5b0-4bd1-b7ee-4946166fed00","Type":"ContainerDied","Data":"7ea5cb132fc5104ad16798402a899bba9015f569ee58b602db21af6913595cd6"}
Nov 02 14:42:17.563859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.563838    4492 scope.go:122] "RemoveContainer" containerID="054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9"
Nov 02 14:42:17.564186 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:17.564166    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9\": container with ID starting with 054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9 not found: ID does not exist" containerID="054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9"
Nov 02 14:42:17.564263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.564198    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9"} err="failed to get container status \"054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9\": rpc error: code = NotFound desc = could not find container \"054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9\": container with ID starting with 054acec5bf4986ea03a793df17a727e186706aee9e48ecf715380addc8974aa9 not found: ID does not exist"
Nov 02 14:42:17.564263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.564223    4492 scope.go:122] "RemoveContainer" containerID="ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4"
Nov 02 14:42:17.573445 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.573421    4492 scope.go:122] "RemoveContainer" containerID="ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4"
Nov 02 14:42:17.573738 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:17.573721    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4\": container with ID starting with ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4 not found: ID does not exist" containerID="ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4"
Nov 02 14:42:17.573796 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.573746    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4"} err="failed to get container status \"ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4\": rpc error: code = NotFound desc = could not find container \"ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4\": container with ID starting with ee183fc0f3a123f0b9d4bda72d2e2a41a17f452de228c3fff4dd2611c83101c4 not found: ID does not exist"
Nov 02 14:42:17.607918 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.607884    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"]
Nov 02 14:42:17.616786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.616758    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-866159/premium-w3-866159-kserve-7dccbdf864-wwb8b"]
Nov 02 14:42:17.656968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.656939    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"]
Nov 02 14:42:17.666869 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.666842    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-866159/simulator-w3-866159-kserve-9bb448ccc-ldxxg"]
Nov 02 14:42:17.925586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:17.925559    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:18.341109 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:18.341021    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:19.315770 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.315712    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445" podUID="87eabb3c-979d-43c2-abb6-dc305fe31238" containerName="payload-pre-processing" containerID="cri-o://b9a454f0d2d775023867e1e0eb5929b3d76cb24c7966610246a7a946f62278fc" gracePeriod=25
Nov 02 14:42:19.572511 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.572481    4492 generic.go:356] "Generic (PLEG): container finished" podID="87eabb3c-979d-43c2-abb6-dc305fe31238" containerID="b9a454f0d2d775023867e1e0eb5929b3d76cb24c7966610246a7a946f62278fc" exitCode=0
Nov 02 14:42:19.572954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.572536    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445" event={"ID":"87eabb3c-979d-43c2-abb6-dc305fe31238","Type":"ContainerDied","Data":"b9a454f0d2d775023867e1e0eb5929b3d76cb24c7966610246a7a946f62278fc"}
Nov 02 14:42:19.572954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.572574    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445" event={"ID":"87eabb3c-979d-43c2-abb6-dc305fe31238","Type":"ContainerDied","Data":"4acb24c3a3cecca540acf02df478480f10daa910a3b1332d8adf1c7f411081b6"}
Nov 02 14:42:19.572954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.572591    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4acb24c3a3cecca540acf02df478480f10daa910a3b1332d8adf1c7f411081b6"
Nov 02 14:42:19.581722 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.581699    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" path="/var/lib/kubelet/pods/26d970fc-38ab-48a8-aab2-984e07c2b1d3/volumes"
Nov 02 14:42:19.582175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.582161    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" path="/var/lib/kubelet/pods/fea0c720-e5b0-4bd1-b7ee-4946166fed00/volumes"
Nov 02 14:42:19.587968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.587949    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:42:19.737676 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.737645    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume\") pod \"87eabb3c-979d-43c2-abb6-dc305fe31238\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") "
Nov 02 14:42:19.737898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.737718    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn\" (UniqueName: \"kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn\") pod \"87eabb3c-979d-43c2-abb6-dc305fe31238\" (UID: \"87eabb3c-979d-43c2-abb6-dc305fe31238\") "
Nov 02 14:42:19.738160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.738047    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume" pod "87eabb3c-979d-43c2-abb6-dc305fe31238" (UID: "87eabb3c-979d-43c2-abb6-dc305fe31238"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:42:19.738600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.738571    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87eabb3c-979d-43c2-abb6-dc305fe31238-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:19.740120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.740098    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn" pod "87eabb3c-979d-43c2-abb6-dc305fe31238" (UID: "87eabb3c-979d-43c2-abb6-dc305fe31238"). InnerVolumeSpecName "kube-api-access-c78mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:19.839853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:19.839778    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78mn\" (UniqueName: \"kubernetes.io/projected/87eabb3c-979d-43c2-abb6-dc305fe31238-kube-api-access-c78mn\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:20.225134 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:20.225099    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:20.344635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:20.344595    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:20.578664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:20.578582    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"
Nov 02 14:42:20.628636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:20.628596    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"]
Nov 02 14:42:20.638396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:20.638349    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-f72bc6-74b59bb48c-j6445"]
Nov 02 14:42:21.582816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:21.582779    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87eabb3c-979d-43c2-abb6-dc305fe31238" path="/var/lib/kubelet/pods/87eabb3c-979d-43c2-abb6-dc305fe31238/volumes"
Nov 02 14:42:22.236760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:22.236713    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:24.049405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:24.049345    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:24.171872 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:24.171834    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:25.183836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:25.183790    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:42:26.060001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:26.059958    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:26.190668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:26.190622    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:42:26.667913 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:26.667874    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:27.632474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:27.632441    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"]
Nov 02 14:42:27.632919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:27.632663    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9" podUID="db27a56f-6686-4998-92d8-43e02b245981" containerName="maas-api" containerID="cri-o://3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c" gracePeriod=30
Nov 02 14:42:27.694333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:27.694300    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:27.887320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:27.887256    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:42:28.054885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.054851    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca\") pod \"db27a56f-6686-4998-92d8-43e02b245981\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") "
Nov 02 14:42:28.055100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.055029    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs\") pod \"db27a56f-6686-4998-92d8-43e02b245981\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") "
Nov 02 14:42:28.055100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.055075    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls\") pod \"db27a56f-6686-4998-92d8-43e02b245981\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") "
Nov 02 14:42:28.055237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.055155    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx\" (UniqueName: \"kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx\") pod \"db27a56f-6686-4998-92d8-43e02b245981\" (UID: \"db27a56f-6686-4998-92d8-43e02b245981\") "
Nov 02 14:42:28.055323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.055294    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca" pod "db27a56f-6686-4998-92d8-43e02b245981" (UID: "db27a56f-6686-4998-92d8-43e02b245981"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:42:28.055675 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.055599    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db27a56f-6686-4998-92d8-43e02b245981-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:28.058284 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.058256    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls" pod "db27a56f-6686-4998-92d8-43e02b245981" (UID: "db27a56f-6686-4998-92d8-43e02b245981"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:42:28.058423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.058390    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx" pod "db27a56f-6686-4998-92d8-43e02b245981" (UID: "db27a56f-6686-4998-92d8-43e02b245981"). InnerVolumeSpecName "kube-api-access-frndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:28.086378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.084251    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs" pod "db27a56f-6686-4998-92d8-43e02b245981" (UID: "db27a56f-6686-4998-92d8-43e02b245981"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:42:28.156248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.156141    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:28.156248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.156170    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db27a56f-6686-4998-92d8-43e02b245981-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:28.156248 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.156181    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frndx\" (UniqueName: \"kubernetes.io/projected/db27a56f-6686-4998-92d8-43e02b245981-kube-api-access-frndx\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:28.455806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.455776    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:28.648393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.648339    4492 generic.go:356] "Generic (PLEG): container finished" podID="db27a56f-6686-4998-92d8-43e02b245981" containerID="3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c" exitCode=0
Nov 02 14:42:28.649023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.648394    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9" event={"ID":"db27a56f-6686-4998-92d8-43e02b245981","Type":"ContainerDied","Data":"3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c"}
Nov 02 14:42:28.649120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.649072    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9" event={"ID":"db27a56f-6686-4998-92d8-43e02b245981","Type":"ContainerDied","Data":"f8148c20cf4f1941cb0f4651b50cbf2324dddf75ae011bbb4f6f0c4b842ec2c8"}
Nov 02 14:42:28.649120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.648428    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"
Nov 02 14:42:28.649240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.649130    4492 scope.go:122] "RemoveContainer" containerID="3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c"
Nov 02 14:42:28.663933 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.663909    4492 scope.go:122] "RemoveContainer" containerID="3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c"
Nov 02 14:42:28.664719 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:28.664691    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c\": container with ID starting with 3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c not found: ID does not exist" containerID="3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c"
Nov 02 14:42:28.664837 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.664730    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c"} err="failed to get container status \"3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c\": rpc error: code = NotFound desc = could not find container \"3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c\": container with ID starting with 3444fbd916449ad45c1f5b9fc4ddb4eb91bbcfd7e86c147066b2b16b8e360e0c not found: ID does not exist"
Nov 02 14:42:28.701100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.701058    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"]
Nov 02 14:42:28.705473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:28.705447    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-866159-6f445bf584-sqfn9"]
Nov 02 14:42:29.580522 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:29.580489    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db27a56f-6686-4998-92d8-43e02b245981" path="/var/lib/kubelet/pods/db27a56f-6686-4998-92d8-43e02b245981/volumes"
Nov 02 14:42:30.195501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:30.195465    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:30.872790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:30.872755    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:31.764714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:31.764679    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:31.908154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:31.908125    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:33.034223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:33.034187    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:33.870695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:33.870656    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:34.006762 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.006723    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:34.127200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.127175    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:42:34.167587 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.167555    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:42:34.221393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.218439    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume\") pod \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") "
Nov 02 14:42:34.221393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.219546    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume" pod "fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" (UID: "fd8e9a7b-0c64-4177-8f58-8be6670c3f2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:42:34.319517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.319480    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume\") pod \"077b9507-47f3-4c34-82c7-27f5a954feb8\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") "
Nov 02 14:42:34.319718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.319542    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw\" (UniqueName: \"kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw\") pod \"077b9507-47f3-4c34-82c7-27f5a954feb8\" (UID: \"077b9507-47f3-4c34-82c7-27f5a954feb8\") "
Nov 02 14:42:34.319718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.319574    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9\" (UniqueName: \"kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9\") pod \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\" (UID: \"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f\") "
Nov 02 14:42:34.319868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.319844    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume" pod "077b9507-47f3-4c34-82c7-27f5a954feb8" (UID: "077b9507-47f3-4c34-82c7-27f5a954feb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:42:34.320002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.319975    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077b9507-47f3-4c34-82c7-27f5a954feb8-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:34.320002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.319994    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:34.321807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.321785    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw" pod "077b9507-47f3-4c34-82c7-27f5a954feb8" (UID: "077b9507-47f3-4c34-82c7-27f5a954feb8"). InnerVolumeSpecName "kube-api-access-mzcdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:34.321919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.321820    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9" pod "fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" (UID: "fd8e9a7b-0c64-4177-8f58-8be6670c3f2f"). InnerVolumeSpecName "kube-api-access-lk9z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:34.421418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.421314    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcdw\" (UniqueName: \"kubernetes.io/projected/077b9507-47f3-4c34-82c7-27f5a954feb8-kube-api-access-mzcdw\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:34.421418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.421343    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9z9\" (UniqueName: \"kubernetes.io/projected/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f-kube-api-access-lk9z9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:34.687058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.687018    4492 generic.go:356] "Generic (PLEG): container finished" podID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerID="f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917" exitCode=0
Nov 02 14:42:34.687257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.687090    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"
Nov 02 14:42:34.687257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.687111    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" event={"ID":"077b9507-47f3-4c34-82c7-27f5a954feb8","Type":"ContainerDied","Data":"f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917"}
Nov 02 14:42:34.687257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.687166    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9" event={"ID":"077b9507-47f3-4c34-82c7-27f5a954feb8","Type":"ContainerDied","Data":"7b416fbc3c2cf4a513d162c5448f55f6bbd96c774b65e48d0c68276af932bbe4"}
Nov 02 14:42:34.687257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.687191    4492 scope.go:122] "RemoveContainer" containerID="f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917"
Nov 02 14:42:34.689018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.688990    4492 generic.go:356] "Generic (PLEG): container finished" podID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerID="c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52" exitCode=0
Nov 02 14:42:34.689126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.689031    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" event={"ID":"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f","Type":"ContainerDied","Data":"c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52"}
Nov 02 14:42:34.689126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.689055    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"
Nov 02 14:42:34.689126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.689068    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx" event={"ID":"fd8e9a7b-0c64-4177-8f58-8be6670c3f2f","Type":"ContainerDied","Data":"4716e5ca7963412a0f55181264511d072b06f5648763a0f900b0ccb4d08c4260"}
Nov 02 14:42:34.698063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.698045    4492 scope.go:122] "RemoveContainer" containerID="f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917"
Nov 02 14:42:34.698367 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:34.698333    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917\": container with ID starting with f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917 not found: ID does not exist" containerID="f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917"
Nov 02 14:42:34.698444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.698378    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917"} err="failed to get container status \"f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917\": rpc error: code = NotFound desc = could not find container \"f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917\": container with ID starting with f44b0cb2d89232e59814cac18606373bdd5322eca7df410497128d9c5851a917 not found: ID does not exist"
Nov 02 14:42:34.698444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.698396    4492 scope.go:122] "RemoveContainer" containerID="c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52"
Nov 02 14:42:34.707340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.707320    4492 scope.go:122] "RemoveContainer" containerID="c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52"
Nov 02 14:42:34.707646 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:34.707627    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52\": container with ID starting with c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52 not found: ID does not exist" containerID="c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52"
Nov 02 14:42:34.707691 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.707654    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52"} err="failed to get container status \"c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52\": rpc error: code = NotFound desc = could not find container \"c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52\": container with ID starting with c11e271f4db729102ed33fbe31ffedae36af9aced615d08e4857144a5114eb52 not found: ID does not exist"
Nov 02 14:42:34.738114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.738075    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"]
Nov 02 14:42:34.745435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.745405    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-866159-7ff65f964c-jz6w9"]
Nov 02 14:42:34.779384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.777480    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"]
Nov 02 14:42:34.781523 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.781503    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-866159-7b94c5b8c9-n76dx"]
Nov 02 14:42:34.995411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:34.995295    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:35.581565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:35.581534    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" path="/var/lib/kubelet/pods/077b9507-47f3-4c34-82c7-27f5a954feb8/volumes"
Nov 02 14:42:35.582106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:35.581959    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" path="/var/lib/kubelet/pods/fd8e9a7b-0c64-4177-8f58-8be6670c3f2f/volumes"
Nov 02 14:42:37.056491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:37.056460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:37.900578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:37.900539    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:38.029712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:38.029675    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:39.021390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:39.021341    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:39.151383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:39.150891    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:39.293329 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:39.293230    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:40.479816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:40.479778    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:40.944970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:40.944934    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:41.468340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:41.468302    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:42.183418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:42.183385    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:42.962420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:42.962381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:43.375882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:43.375799    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:43.778474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:43.778433    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"]
Nov 02 14:42:43.778697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:43.778673    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy" containerID="cri-o://d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73" gracePeriod=30
Nov 02 14:42:43.817554 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:43.817517    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:43.944498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:43.944463    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:44.773161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:44.773131    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:45.161732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:45.161649    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:45.311663 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:45.311624    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:46.516936 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:46.516898    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:42:47.022145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:47.022115    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:48.718297 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:48.718267    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:48.847382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:48.847331    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:48.973387 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:48.973289    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:49.060777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.060750    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:42:49.231979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.231898    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.231997    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232048    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232076    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232126    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232160    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232181    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:49.232412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232192    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232313    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:49.232412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232340    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232410    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo\") pod \"b0912ef0-0fcb-4dfb-8c36-73e809947187\" (UID: \"b0912ef0-0fcb-4dfb-8c36-73e809947187\") "
Nov 02 14:42:49.232705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232451    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:49.232763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232728    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:42:49.232835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.232786    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:49.233106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.233044    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.233106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.233070    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.233106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.233085    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.233106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.233110    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0912ef0-0fcb-4dfb-8c36-73e809947187-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.233106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.233123    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.234585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.234560    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:42:49.234763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.234719    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:42:49.234763 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.234753    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:49.234928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.234749    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs" pod "b0912ef0-0fcb-4dfb-8c36-73e809947187" (UID: "b0912ef0-0fcb-4dfb-8c36-73e809947187"). InnerVolumeSpecName "kube-api-access-8jdxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:49.273859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.273815    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:49.334577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.334538    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.334577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.334575    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.334783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.334589    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdxs\" (UniqueName: \"kubernetes.io/projected/b0912ef0-0fcb-4dfb-8c36-73e809947187-kube-api-access-8jdxs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.334783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.334603    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0912ef0-0fcb-4dfb-8c36-73e809947187-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:49.385709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.385674    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:49.820594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.820555    4492 generic.go:356] "Generic (PLEG): container finished" podID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerID="d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73" exitCode=0
Nov 02 14:42:49.821130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.820614    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"
Nov 02 14:42:49.821130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.820651    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" event={"ID":"b0912ef0-0fcb-4dfb-8c36-73e809947187","Type":"ContainerDied","Data":"d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73"}
Nov 02 14:42:49.821130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.820723    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m" event={"ID":"b0912ef0-0fcb-4dfb-8c36-73e809947187","Type":"ContainerDied","Data":"cbfd0ac30fc04c678692de7dd2a836fd83caba0a4651574da0b08cf1186c2b6e"}
Nov 02 14:42:49.821130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.820753    4492 scope.go:122] "RemoveContainer" containerID="d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73"
Nov 02 14:42:49.831643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.831617    4492 scope.go:122] "RemoveContainer" containerID="d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73"
Nov 02 14:42:49.831977 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:49.831954    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73\": container with ID starting with d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73 not found: ID does not exist" containerID="d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73"
Nov 02 14:42:49.832033 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.831986    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73"} err="failed to get container status \"d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73\": rpc error: code = NotFound desc = could not find container \"d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73\": container with ID starting with d928e217fd291bf3ace2460fb453a5e2e9660dff2d535c4cfb7a300c41576a73 not found: ID does not exist"
Nov 02 14:42:49.878015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.877925    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"]
Nov 02 14:42:49.887850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:49.887812    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5r78m"]
Nov 02 14:42:50.849237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:50.849206    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:51.587755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:51.587713    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" path="/var/lib/kubelet/pods/b0912ef0-0fcb-4dfb-8c36-73e809947187/volumes"
Nov 02 14:42:52.557843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:52.557799    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"]
Nov 02 14:42:52.591783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:52.591751    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"]
Nov 02 14:42:52.727853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:52.727824    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:52.843741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:52.843637    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:52.985303 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:52.985228    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:53.561009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:53.560977    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:54.839021 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:54.838980    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:56.435449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.435231    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"]
Nov 02 14:42:56.436336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.436293    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr" podUID="911d6ffa-04a7-4547-b669-4351aa9f2533" containerName="maas-api" containerID="cri-o://f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4" gracePeriod=30
Nov 02 14:42:56.488952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.488913    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:56.738418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.738394    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:42:56.813701 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.813659    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca\") pod \"911d6ffa-04a7-4547-b669-4351aa9f2533\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") "
Nov 02 14:42:56.813935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.813728    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76\" (UniqueName: \"kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76\") pod \"911d6ffa-04a7-4547-b669-4351aa9f2533\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") "
Nov 02 14:42:56.813935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.813771    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs\") pod \"911d6ffa-04a7-4547-b669-4351aa9f2533\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") "
Nov 02 14:42:56.813935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.813846    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls\") pod \"911d6ffa-04a7-4547-b669-4351aa9f2533\" (UID: \"911d6ffa-04a7-4547-b669-4351aa9f2533\") "
Nov 02 14:42:56.814185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.814154    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca" pod "911d6ffa-04a7-4547-b669-4351aa9f2533" (UID: "911d6ffa-04a7-4547-b669-4351aa9f2533"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:42:56.814333 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.814313    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911d6ffa-04a7-4547-b669-4351aa9f2533-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:56.816290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.816257    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls" pod "911d6ffa-04a7-4547-b669-4351aa9f2533" (UID: "911d6ffa-04a7-4547-b669-4351aa9f2533"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:42:56.816581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.816557    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs" pod "911d6ffa-04a7-4547-b669-4351aa9f2533" (UID: "911d6ffa-04a7-4547-b669-4351aa9f2533"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:42:56.816692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.816591    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76" pod "911d6ffa-04a7-4547-b669-4351aa9f2533" (UID: "911d6ffa-04a7-4547-b669-4351aa9f2533"). InnerVolumeSpecName "kube-api-access-mcj76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:42:56.867973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.867938    4492 generic.go:356] "Generic (PLEG): container finished" podID="911d6ffa-04a7-4547-b669-4351aa9f2533" containerID="f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4" exitCode=0
Nov 02 14:42:56.868143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.867986    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"
Nov 02 14:42:56.868143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.868027    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr" event={"ID":"911d6ffa-04a7-4547-b669-4351aa9f2533","Type":"ContainerDied","Data":"f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4"}
Nov 02 14:42:56.868143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.868085    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr" event={"ID":"911d6ffa-04a7-4547-b669-4351aa9f2533","Type":"ContainerDied","Data":"07eca7b9753f08f55bce45d2d4289cc2ff0a2c2943f3dbaff39b54b13f2c605c"}
Nov 02 14:42:56.868143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.868112    4492 scope.go:122] "RemoveContainer" containerID="f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4"
Nov 02 14:42:56.878378 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.878332    4492 scope.go:122] "RemoveContainer" containerID="f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4"
Nov 02 14:42:56.878710 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:42:56.878690    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4\": container with ID starting with f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4 not found: ID does not exist" containerID="f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4"
Nov 02 14:42:56.878769 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.878718    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4"} err="failed to get container status \"f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4\": rpc error: code = NotFound desc = could not find container \"f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4\": container with ID starting with f8cd902be9c898f37874193103296931fecdbbcd26b66e9fbb52c4175aaea5e4 not found: ID does not exist"
Nov 02 14:42:56.929612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.929484    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:56.929612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.929514    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcj76\" (UniqueName: \"kubernetes.io/projected/911d6ffa-04a7-4547-b669-4351aa9f2533-kube-api-access-mcj76\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:56.929612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.929527    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/911d6ffa-04a7-4547-b669-4351aa9f2533-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:42:56.932855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.932825    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"]
Nov 02 14:42:56.935170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:56.935144    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bf0b0d-c575c64c-zm4hr"]
Nov 02 14:42:57.156944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:57.156862    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:42:57.558263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:57.558213    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerName="payload-processing" containerID="cri-o://d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a" gracePeriod=25
Nov 02 14:42:57.579556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:57.579516    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911d6ffa-04a7-4547-b669-4351aa9f2533" path="/var/lib/kubelet/pods/911d6ffa-04a7-4547-b669-4351aa9f2533/volumes"
Nov 02 14:42:57.592392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:57.592340    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" containerName="payload-pre-processing" containerID="cri-o://fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c" gracePeriod=25
Nov 02 14:42:59.047238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:42:59.047190    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:00.616843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:00.616785    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:00.734608 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:00.734577    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:00.947428 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:00.947397    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:01.465069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:01.465045    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:02.015622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:02.015564    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:03.014416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:03.014338    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:03.029430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:03.029393    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:03.029843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:03.029406    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:03.135039 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:03.135007    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:04.840383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:04.840314    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:06.734658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:06.734626    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:08.645277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:08.645244    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:09.350766 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:09.350736    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"]
Nov 02 14:43:09.383500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:09.383468    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"]
Nov 02 14:43:09.501534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:09.501507    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:09.619060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:09.618963    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:10.255384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.255316    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"]
Nov 02 14:43:10.255877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.255635    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b" podUID="785248a9-5836-4d30-a522-0be624cf1b3f" containerName="maas-api" containerID="cri-o://f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6" gracePeriod=30
Nov 02 14:43:10.414892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.414815    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:10.538429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.532450    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:10.538429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.534074    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:43:10.617897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.617869    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca\") pod \"785248a9-5836-4d30-a522-0be624cf1b3f\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") "
Nov 02 14:43:10.618093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.617929    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr\" (UniqueName: \"kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr\") pod \"785248a9-5836-4d30-a522-0be624cf1b3f\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") "
Nov 02 14:43:10.618226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.618132    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls\") pod \"785248a9-5836-4d30-a522-0be624cf1b3f\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") "
Nov 02 14:43:10.618226 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.618174    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs\") pod \"785248a9-5836-4d30-a522-0be624cf1b3f\" (UID: \"785248a9-5836-4d30-a522-0be624cf1b3f\") "
Nov 02 14:43:10.618339 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.618288    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca" pod "785248a9-5836-4d30-a522-0be624cf1b3f" (UID: "785248a9-5836-4d30-a522-0be624cf1b3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:10.618616 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.618595    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/785248a9-5836-4d30-a522-0be624cf1b3f-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:10.620307 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.620281    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls" pod "785248a9-5836-4d30-a522-0be624cf1b3f" (UID: "785248a9-5836-4d30-a522-0be624cf1b3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:43:10.620434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.620299    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr" pod "785248a9-5836-4d30-a522-0be624cf1b3f" (UID: "785248a9-5836-4d30-a522-0be624cf1b3f"). InnerVolumeSpecName "kube-api-access-bz2wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:10.620434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.620301    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs" pod "785248a9-5836-4d30-a522-0be624cf1b3f" (UID: "785248a9-5836-4d30-a522-0be624cf1b3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:43:10.645800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.645774    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:10.729381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.719189    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2wr\" (UniqueName: \"kubernetes.io/projected/785248a9-5836-4d30-a522-0be624cf1b3f-kube-api-access-bz2wr\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:10.729381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.719220    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:10.729381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.719229    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/785248a9-5836-4d30-a522-0be624cf1b3f-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:10.955544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.955508    4492 generic.go:356] "Generic (PLEG): container finished" podID="785248a9-5836-4d30-a522-0be624cf1b3f" containerID="f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6" exitCode=0
Nov 02 14:43:10.955716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.955552    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"
Nov 02 14:43:10.955716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.955588    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b" event={"ID":"785248a9-5836-4d30-a522-0be624cf1b3f","Type":"ContainerDied","Data":"f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6"}
Nov 02 14:43:10.955716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.955644    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b" event={"ID":"785248a9-5836-4d30-a522-0be624cf1b3f","Type":"ContainerDied","Data":"2aa077f5ee09613d5049993ede417d2325d690590c3c9e3acfbe1b06338594cc"}
Nov 02 14:43:10.955716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.955669    4492 scope.go:122] "RemoveContainer" containerID="f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6"
Nov 02 14:43:10.966393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.966371    4492 scope.go:122] "RemoveContainer" containerID="f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6"
Nov 02 14:43:10.966709 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:10.966689    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6\": container with ID starting with f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6 not found: ID does not exist" containerID="f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6"
Nov 02 14:43:10.966791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:10.966718    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6"} err="failed to get container status \"f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6\": rpc error: code = NotFound desc = could not find container \"f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6\": container with ID starting with f5bdc77132c80a26cce312abea55e493afbfdc8284699cb1fa7dea3d375fc2e6 not found: ID does not exist"
Nov 02 14:43:11.007110 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:11.007084    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"]
Nov 02 14:43:11.011332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:11.011303    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2bb4ef-55864b9d4c-rb42b"]
Nov 02 14:43:11.579698 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:11.579666    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785248a9-5836-4d30-a522-0be624cf1b3f" path="/var/lib/kubelet/pods/785248a9-5836-4d30-a522-0be624cf1b3f/volumes"
Nov 02 14:43:13.029121 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:13.029072    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:13.029557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:13.029090    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:14.351702 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:14.351615    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerName="payload-processing" containerID="cri-o://eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5" gracePeriod=25
Nov 02 14:43:14.384626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:14.384582    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" containerName="payload-pre-processing" containerID="cri-o://9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea" gracePeriod=25
Nov 02 14:43:16.516052 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:16.516015    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:43:17.091994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.091960    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:17.205099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.205047    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:17.275498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.275463    4492 scope.go:122] "RemoveContainer" containerID="b9a454f0d2d775023867e1e0eb5929b3d76cb24c7966610246a7a946f62278fc"
Nov 02 14:43:17.854723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.854647    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:17.859044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.859018    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:43:17.875422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.875385    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:43:17.904381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.904138    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume\") pod \"256171e0-9920-4309-bb65-8cee75fac1ed\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") "
Nov 02 14:43:17.905141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.905100    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume" pod "256171e0-9920-4309-bb65-8cee75fac1ed" (UID: "256171e0-9920-4309-bb65-8cee75fac1ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:17.916062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.908164    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume\") pod \"100f4629-f879-4c79-aa08-71cbb4ad864d\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") "
Nov 02 14:43:17.916062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.908246    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh\" (UniqueName: \"kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh\") pod \"100f4629-f879-4c79-aa08-71cbb4ad864d\" (UID: \"100f4629-f879-4c79-aa08-71cbb4ad864d\") "
Nov 02 14:43:17.916062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.908291    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f\") pod \"256171e0-9920-4309-bb65-8cee75fac1ed\" (UID: \"256171e0-9920-4309-bb65-8cee75fac1ed\") "
Nov 02 14:43:17.916062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.908834    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256171e0-9920-4309-bb65-8cee75fac1ed-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:17.916062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.914048    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume" pod "100f4629-f879-4c79-aa08-71cbb4ad864d" (UID: "100f4629-f879-4c79-aa08-71cbb4ad864d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:17.923172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.923135    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f" pod "256171e0-9920-4309-bb65-8cee75fac1ed" (UID: "256171e0-9920-4309-bb65-8cee75fac1ed"). InnerVolumeSpecName "kube-api-access-5x45f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:17.923390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:17.923182    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh" pod "100f4629-f879-4c79-aa08-71cbb4ad864d" (UID: "100f4629-f879-4c79-aa08-71cbb4ad864d"). InnerVolumeSpecName "kube-api-access-pvpkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:18.008569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.008528    4492 generic.go:356] "Generic (PLEG): container finished" podID="256171e0-9920-4309-bb65-8cee75fac1ed" containerID="fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c" exitCode=0
Nov 02 14:43:18.008569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.008575    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"
Nov 02 14:43:18.008826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.008589    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" event={"ID":"256171e0-9920-4309-bb65-8cee75fac1ed","Type":"ContainerDied","Data":"fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c"}
Nov 02 14:43:18.008826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.008653    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm" event={"ID":"256171e0-9920-4309-bb65-8cee75fac1ed","Type":"ContainerDied","Data":"a53152738acaf68a06d1ac066daa849b15a1d4fcffce3811b21d00d53c0615af"}
Nov 02 14:43:18.008826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.008684    4492 scope.go:122] "RemoveContainer" containerID="fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c"
Nov 02 14:43:18.009523 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.009501    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f4629-f879-4c79-aa08-71cbb4ad864d-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:18.009523 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.009523    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpkh\" (UniqueName: \"kubernetes.io/projected/100f4629-f879-4c79-aa08-71cbb4ad864d-kube-api-access-pvpkh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:18.009741 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.009539    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x45f\" (UniqueName: \"kubernetes.io/projected/256171e0-9920-4309-bb65-8cee75fac1ed-kube-api-access-5x45f\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:18.010250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.010227    4492 generic.go:356] "Generic (PLEG): container finished" podID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerID="d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a" exitCode=0
Nov 02 14:43:18.010490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.010320    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"
Nov 02 14:43:18.010490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.010332    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" event={"ID":"100f4629-f879-4c79-aa08-71cbb4ad864d","Type":"ContainerDied","Data":"d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a"}
Nov 02 14:43:18.010490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.010407    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl" event={"ID":"100f4629-f879-4c79-aa08-71cbb4ad864d","Type":"ContainerDied","Data":"6b43cd229d10a9928142d5a5553dc5887557455949fdec18194fa26e6a350b8f"}
Nov 02 14:43:18.019912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.019887    4492 scope.go:122] "RemoveContainer" containerID="fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c"
Nov 02 14:43:18.020286 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:18.020249    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c\": container with ID starting with fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c not found: ID does not exist" containerID="fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c"
Nov 02 14:43:18.020400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.020301    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c"} err="failed to get container status \"fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c\": rpc error: code = NotFound desc = could not find container \"fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c\": container with ID starting with fa7c4d5a0aea79355bf3f3b599bebaed12b66767c361438f7e019fd908b3739c not found: ID does not exist"
Nov 02 14:43:18.020400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.020330    4492 scope.go:122] "RemoveContainer" containerID="d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a"
Nov 02 14:43:18.029847 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.029822    4492 scope.go:122] "RemoveContainer" containerID="d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a"
Nov 02 14:43:18.030074 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:18.030055    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a\": container with ID starting with d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a not found: ID does not exist" containerID="d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a"
Nov 02 14:43:18.030129 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.030080    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a"} err="failed to get container status \"d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a\": rpc error: code = NotFound desc = could not find container \"d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a\": container with ID starting with d05aa2515b77fd5c9af535d3d5f8a64694be1433ed7fa600eb9d8cb47154df6a not found: ID does not exist"
Nov 02 14:43:18.035061 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:18.035032    4492 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256171e0_9920_4309_bb65_8cee75fac1ed.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256171e0_9920_4309_bb65_8cee75fac1ed.slice/crio-a53152738acaf68a06d1ac066daa849b15a1d4fcffce3811b21d00d53c0615af\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f4629_f879_4c79_aa08_71cbb4ad864d.slice/crio-6b43cd229d10a9928142d5a5553dc5887557455949fdec18194fa26e6a350b8f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f4629_f879_4c79_aa08_71cbb4ad864d.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 14:43:18.073194 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.073158    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"]
Nov 02 14:43:18.077726 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.077703    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bf0b0d-58bbc75bb7-zp2nm"]
Nov 02 14:43:18.111012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.110917    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"]
Nov 02 14:43:18.116966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.116932    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bf0b0d-84686fb7dc-wccfl"]
Nov 02 14:43:18.290626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.290590    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"]
Nov 02 14:43:18.291113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.291077    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy" containerID="cri-o://86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c" gracePeriod=30
Nov 02 14:43:18.367915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:18.367812    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:19.410550 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:19.410504    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:19.580345 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:19.580306    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" path="/var/lib/kubelet/pods/100f4629-f879-4c79-aa08-71cbb4ad864d/volumes"
Nov 02 14:43:19.580755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:19.580732    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" path="/var/lib/kubelet/pods/256171e0-9920-4309-bb65-8cee75fac1ed/volumes"
Nov 02 14:43:20.387521 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:20.387483    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:20.416381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:20.416314    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:23.547863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.547838    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:43:23.567865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.567809    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.567865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.567853    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.567918    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.567944    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.567975    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568075    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568130    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568170    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568209    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert\") pod \"e58065a4-78ba-4569-81f7-5f96483077d9\" (UID: \"e58065a4-78ba-4569-81f7-5f96483077d9\") "
Nov 02 14:43:23.568295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568229    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:23.568295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568282    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:23.568494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568388    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:23.568692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568672    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.568771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568697    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.568771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568710    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.568771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568678    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:23.568890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.568857    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:23.570047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.570029    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "kube-api-access-d76sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:23.570179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.570161    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:43:23.570651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.570632    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:23.570717 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.570655    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy" pod "e58065a4-78ba-4569-81f7-5f96483077d9" (UID: "e58065a4-78ba-4569-81f7-5f96483077d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:23.669622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.669590    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.669622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.669618    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58065a4-78ba-4569-81f7-5f96483077d9-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.669622 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.669628    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58065a4-78ba-4569-81f7-5f96483077d9-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.669864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.669636    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d76sc\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-kube-api-access-d76sc\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.669864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.669645    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58065a4-78ba-4569-81f7-5f96483077d9-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:23.669864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:23.669652    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58065a4-78ba-4569-81f7-5f96483077d9-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:24.044539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.044508    4492 generic.go:356] "Generic (PLEG): container finished" podID="e58065a4-78ba-4569-81f7-5f96483077d9" containerID="86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c" exitCode=0
Nov 02 14:43:24.044737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.044559    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"
Nov 02 14:43:24.044737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.044565    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" event={"ID":"e58065a4-78ba-4569-81f7-5f96483077d9","Type":"ContainerDied","Data":"86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c"}
Nov 02 14:43:24.044737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.044692    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4" event={"ID":"e58065a4-78ba-4569-81f7-5f96483077d9","Type":"ContainerDied","Data":"17f78a6059692559c39e454b37635d8647c1419734a93945c9df07ab42409b61"}
Nov 02 14:43:24.044846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.044748    4492 scope.go:122] "RemoveContainer" containerID="86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c"
Nov 02 14:43:24.054179 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.054157    4492 scope.go:122] "RemoveContainer" containerID="86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c"
Nov 02 14:43:24.054505 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:24.054482    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c\": container with ID starting with 86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c not found: ID does not exist" containerID="86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c"
Nov 02 14:43:24.054600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.054518    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c"} err="failed to get container status \"86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c\": rpc error: code = NotFound desc = could not find container \"86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c\": container with ID starting with 86260d9aa71d60c4811afe231cee618d83d0cfea9b1512f51c827324d03a341c not found: ID does not exist"
Nov 02 14:43:24.102584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.102550    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"]
Nov 02 14:43:24.106672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:24.106651    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2bb4ef-openshift-default-5bf7b7c64f-2vhb4"]
Nov 02 14:43:25.579269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:25.579234    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" path="/var/lib/kubelet/pods/e58065a4-78ba-4569-81f7-5f96483077d9/volumes"
Nov 02 14:43:29.409871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:29.409821    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:29.786600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:29.786566    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbpwl_c44cd0a2-9688-4f95-8590-7b095ebc93b8/manager/0.log"
Nov 02 14:43:29.996926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:29.996900    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jfj67_28a0d1ac-075a-43b5-8319-02c228b1fa6a/manager/0.log"
Nov 02 14:43:30.416700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:30.416662    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:43:30.693753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:30.693718    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4flwn_3918932b-be47-42ad-8685-9bc895afe0f3/manager/0.log"
Nov 02 14:43:31.190892 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:31.190859    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9dxhn_dd12c716-3e42-4130-a442-dc7b3066ee12/manager/0.log"
Nov 02 14:43:32.697958 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:32.697925    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p5lx_a1fe8ffd-d93b-42a0-a727-88d122676cbc/manager/0.log"
Nov 02 14:43:34.642102 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.642075    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:43:34.671401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.671372    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:43:34.782060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.782033    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j\" (UniqueName: \"kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j\") pod \"38e68747-c49e-4767-bc47-f92c0509aecd\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") "
Nov 02 14:43:34.782251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.782139    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc\" (UniqueName: \"kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc\") pod \"58f4ff8b-716f-4c46-8382-d2c762207fb1\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") "
Nov 02 14:43:34.782251 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.782187    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume\") pod \"38e68747-c49e-4767-bc47-f92c0509aecd\" (UID: \"38e68747-c49e-4767-bc47-f92c0509aecd\") "
Nov 02 14:43:34.782400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.782289    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume\") pod \"58f4ff8b-716f-4c46-8382-d2c762207fb1\" (UID: \"58f4ff8b-716f-4c46-8382-d2c762207fb1\") "
Nov 02 14:43:34.782681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.782650    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume" pod "38e68747-c49e-4767-bc47-f92c0509aecd" (UID: "38e68747-c49e-4767-bc47-f92c0509aecd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:34.782681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.782665    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume" pod "58f4ff8b-716f-4c46-8382-d2c762207fb1" (UID: "58f4ff8b-716f-4c46-8382-d2c762207fb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:34.784293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.784272    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j" pod "38e68747-c49e-4767-bc47-f92c0509aecd" (UID: "38e68747-c49e-4767-bc47-f92c0509aecd"). InnerVolumeSpecName "kube-api-access-48b6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:34.784404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.784338    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc" pod "58f4ff8b-716f-4c46-8382-d2c762207fb1" (UID: "58f4ff8b-716f-4c46-8382-d2c762207fb1"). InnerVolumeSpecName "kube-api-access-9cvtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:34.883953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.883876    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvtc\" (UniqueName: \"kubernetes.io/projected/58f4ff8b-716f-4c46-8382-d2c762207fb1-kube-api-access-9cvtc\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:34.883953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.883904    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e68747-c49e-4767-bc47-f92c0509aecd-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:34.883953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.883913    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f4ff8b-716f-4c46-8382-d2c762207fb1-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:34.883953 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:34.883924    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48b6j\" (UniqueName: \"kubernetes.io/projected/38e68747-c49e-4767-bc47-f92c0509aecd-kube-api-access-48b6j\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:43:35.105312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.105273    4492 generic.go:356] "Generic (PLEG): container finished" podID="38e68747-c49e-4767-bc47-f92c0509aecd" containerID="9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea" exitCode=0
Nov 02 14:43:35.105508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.105317    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"
Nov 02 14:43:35.105508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.105380    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" event={"ID":"38e68747-c49e-4767-bc47-f92c0509aecd","Type":"ContainerDied","Data":"9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea"}
Nov 02 14:43:35.105508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.105436    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6" event={"ID":"38e68747-c49e-4767-bc47-f92c0509aecd","Type":"ContainerDied","Data":"02642c97da688adb779992bd3a68287c45ea82baef8b7045939121681d5f9386"}
Nov 02 14:43:35.105508 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.105452    4492 scope.go:122] "RemoveContainer" containerID="9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea"
Nov 02 14:43:35.106959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.106934    4492 generic.go:356] "Generic (PLEG): container finished" podID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerID="eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5" exitCode=0
Nov 02 14:43:35.107073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.106983    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"
Nov 02 14:43:35.107158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.107135    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" event={"ID":"58f4ff8b-716f-4c46-8382-d2c762207fb1","Type":"ContainerDied","Data":"eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5"}
Nov 02 14:43:35.107278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.107176    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk" event={"ID":"58f4ff8b-716f-4c46-8382-d2c762207fb1","Type":"ContainerDied","Data":"7a8f38f5dfd5adf50797a7ec3cb9a257ad716fe97b0db9c02a1f7c411a5fe1df"}
Nov 02 14:43:35.114886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.114863    4492 scope.go:122] "RemoveContainer" containerID="9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea"
Nov 02 14:43:35.115210 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:35.115179    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea\": container with ID starting with 9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea not found: ID does not exist" containerID="9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea"
Nov 02 14:43:35.115305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.115221    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea"} err="failed to get container status \"9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea\": rpc error: code = NotFound desc = could not find container \"9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea\": container with ID starting with 9ef3e67ee0cb855fc5591da4043e35421ccbec2f8a58919abd8ec1a7156c10ea not found: ID does not exist"
Nov 02 14:43:35.115305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.115245    4492 scope.go:122] "RemoveContainer" containerID="eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5"
Nov 02 14:43:35.124040 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.124022    4492 scope.go:122] "RemoveContainer" containerID="eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5"
Nov 02 14:43:35.124299 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:35.124280    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5\": container with ID starting with eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5 not found: ID does not exist" containerID="eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5"
Nov 02 14:43:35.124348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.124308    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5"} err="failed to get container status \"eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5\": rpc error: code = NotFound desc = could not find container \"eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5\": container with ID starting with eeee92673c8f6eb90ff23e593c9285eb3ee812bff0a02aa58bf1c4e8fdaeedc5 not found: ID does not exist"
Nov 02 14:43:35.165860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.165777    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"]
Nov 02 14:43:35.171280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.171260    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2bb4ef-c686dbc8c-xp6sk"]
Nov 02 14:43:35.201804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.201775    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"]
Nov 02 14:43:35.207932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.207908    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bb4ef-b4ff578f9-lhcd6"]
Nov 02 14:43:35.579124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.579090    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" path="/var/lib/kubelet/pods/38e68747-c49e-4767-bc47-f92c0509aecd/volumes"
Nov 02 14:43:35.579515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:35.579500    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" path="/var/lib/kubelet/pods/58f4ff8b-716f-4c46-8382-d2c762207fb1/volumes"
Nov 02 14:43:40.605730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:40.605695    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:44.908228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908189    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-jdk62"]
Nov 02 14:43:44.908830 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908809    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e1d9f9ec-f170-4d5e-973c-7e03510570ab" containerName="payload-pre-processing"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908833    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d9f9ec-f170-4d5e-973c-7e03510570ab" containerName="payload-pre-processing"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908851    4492 cpu_manager.go:414] "containerMap: removing container" podUID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" containerName="main"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908858    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" containerName="main"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908870    4492 cpu_manager.go:414] "containerMap: removing container" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerName="payload-processing"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908877    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerName="payload-processing"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908893    4492 cpu_manager.go:414] "containerMap: removing container" podUID="785248a9-5836-4d30-a522-0be624cf1b3f" containerName="maas-api"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908901    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785248a9-5836-4d30-a522-0be624cf1b3f" containerName="maas-api"
Nov 02 14:43:44.908912 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908909    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ab230828-8405-43cf-82c9-ff18683d733e" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908917    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab230828-8405-43cf-82c9-ff18683d733e" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908926    4492 cpu_manager.go:414] "containerMap: removing container" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908933    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908945    4492 cpu_manager.go:414] "containerMap: removing container" podUID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" containerName="main"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908951    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" containerName="main"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908963    4492 cpu_manager.go:414] "containerMap: removing container" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908970    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908983    4492 cpu_manager.go:414] "containerMap: removing container" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.908989    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909010    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909016    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909026    4492 cpu_manager.go:414] "containerMap: removing container" podUID="87eabb3c-979d-43c2-abb6-dc305fe31238" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909052    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87eabb3c-979d-43c2-abb6-dc305fe31238" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909071    4492 cpu_manager.go:414] "containerMap: removing container" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909079    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909089    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909096    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerName="payload-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909108    4492 cpu_manager.go:414] "containerMap: removing container" podUID="911d6ffa-04a7-4547-b669-4351aa9f2533" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909115    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911d6ffa-04a7-4547-b669-4351aa9f2533" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909129    4492 cpu_manager.go:414] "containerMap: removing container" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909138    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909152    4492 cpu_manager.go:414] "containerMap: removing container" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909161    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909175    4492 cpu_manager.go:414] "containerMap: removing container" podUID="bfe72977-06e8-46e5-9c35-cb99090fd84a" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909181    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe72977-06e8-46e5-9c35-cb99090fd84a" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909191    4492 cpu_manager.go:414] "containerMap: removing container" podUID="db27a56f-6686-4998-92d8-43e02b245981" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909198    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db27a56f-6686-4998-92d8-43e02b245981" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909206    4492 cpu_manager.go:414] "containerMap: removing container" podUID="8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909213    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" containerName="maas-api"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909223    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909230    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909317    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8e9a7b-0c64-4177-8f58-8be6670c3f2f" containerName="payload-pre-processing"
Nov 02 14:43:44.909321 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909330    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe72977-06e8-46e5-9c35-cb99090fd84a" containerName="maas-api"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909340    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="db27a56f-6686-4998-92d8-43e02b245981" containerName="maas-api"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909348    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae0bf9a-fd3a-4399-b69f-bef6f8f5b7f4" containerName="maas-api"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909374    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0912ef0-0fcb-4dfb-8c36-73e809947187" containerName="istio-proxy"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909383    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d9f9ec-f170-4d5e-973c-7e03510570ab" containerName="payload-pre-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909394    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d970fc-38ab-48a8-aab2-984e07c2b1d3" containerName="main"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909400    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f4ff8b-716f-4c46-8382-d2c762207fb1" containerName="payload-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909410    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="785248a9-5836-4d30-a522-0be624cf1b3f" containerName="maas-api"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909417    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab230828-8405-43cf-82c9-ff18683d733e" containerName="payload-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909426    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e528bec-5bc1-4638-9549-d6bbbdc486af" containerName="payload-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909433    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea0c720-e5b0-4bd1-b7ee-4946166fed00" containerName="main"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909440    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="100f4629-f879-4c79-aa08-71cbb4ad864d" containerName="payload-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909449    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e68747-c49e-4767-bc47-f92c0509aecd" containerName="payload-pre-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909460    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58065a4-78ba-4569-81f7-5f96483077d9" containerName="istio-proxy"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909468    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="87eabb3c-979d-43c2-abb6-dc305fe31238" containerName="payload-pre-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909478    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="077b9507-47f3-4c34-82c7-27f5a954feb8" containerName="payload-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909486    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c683bc-3642-4ce9-84e5-cb88f26e87b4" containerName="payload-processing"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909493    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="911d6ffa-04a7-4547-b669-4351aa9f2533" containerName="maas-api"
Nov 02 14:43:44.910941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.909502    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="256171e0-9920-4309-bb65-8cee75fac1ed" containerName="payload-pre-processing"
Nov 02 14:43:44.912928 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.912908    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:44.928239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.928218    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftwbw\""
Nov 02 14:43:44.931384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.931341    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-jdk62"]
Nov 02 14:43:44.938498 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.938470    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 14:43:44.938642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.938488    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 14:43:44.938642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:44.938570    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 14:43:45.026642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.026607    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.026822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.026651    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544ts\" (UniqueName: \"kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.026822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.026786    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.128241 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.128200    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.128467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.128249    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544ts\" (UniqueName: \"kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.128467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.128299    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.130922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.130898    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.131049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.130898    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.149975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.149945    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544ts\" (UniqueName: \"kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts\") pod \"maas-controller-5db8dc9fd8-jdk62\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") " pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.225853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.225820    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:45.530495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.530407    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-jdk62"]
Nov 02 14:43:45.535215 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:43:45.535174    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e0f510_4f38_4a33_9efb_5c18b53375b9.slice/crio-f3684b44be4d8489e189a50d9b04fc4af59263e12789b86ae893b07645659852 WatchSource:0}: Error finding container f3684b44be4d8489e189a50d9b04fc4af59263e12789b86ae893b07645659852: Status 404 returned error can't find the container with id f3684b44be4d8489e189a50d9b04fc4af59263e12789b86ae893b07645659852
Nov 02 14:43:45.536444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:45.536427    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:43:46.189540 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:46.189494    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62" event={"ID":"96e0f510-4f38-4a33-9efb-5c18b53375b9","Type":"ContainerStarted","Data":"f3684b44be4d8489e189a50d9b04fc4af59263e12789b86ae893b07645659852"}
Nov 02 14:43:46.518889 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:43:46.518750    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:43:48.205278 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:48.205247    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62" event={"ID":"96e0f510-4f38-4a33-9efb-5c18b53375b9","Type":"ContainerStarted","Data":"8afd635d58fa8bf0d402bdba4db7fd1dcd1aa6f6b2dd56d16103ce871faf38ea"}
Nov 02 14:43:48.205708 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:48.205329    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:43:48.244966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:48.244913    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62" podStartSLOduration=2.453826724 podStartE2EDuration="4.244897175s" totalImagesPullingTime="1.791070451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:43:48.242534241 +0000 UTC m=+2792.786249872" watchObservedRunningTime="2026-11-02 14:43:48.244897175 +0000 UTC m=+2792.788612791"
Nov 02 14:43:58.224107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:58.224075    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:43:59.214101 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:43:59.214071    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:44:05.313707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.313670    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"]
Nov 02 14:44:05.317465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.317446    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.333315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.333278    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:44:05.336701 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.336674    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"]
Nov 02 14:44:05.343274 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.343253    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:44:05.349910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.349889    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:44:05.350063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.349963    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:44:05.440558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.440521    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.440739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.440653    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxhf\" (UniqueName: \"kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.440739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.440682    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.541266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.541230    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxhf\" (UniqueName: \"kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.541266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.541271    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.541538 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.541334    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.542030 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.542009    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.544164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.544123    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.563600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.563575    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxhf\" (UniqueName: \"kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf\") pod \"maas-api-revoke-sub-85237e45-4khvt\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.630799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.630701    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:05.931265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:05.931234    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"]
Nov 02 14:44:05.935461 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:44:05.935418    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc101b13_4831_47cc_ae27_f6533d5b360c.slice/crio-fc5211831e89605a188727d42347e904e48246328fc0f341550390e88b4497ac WatchSource:0}: Error finding container fc5211831e89605a188727d42347e904e48246328fc0f341550390e88b4497ac: Status 404 returned error can't find the container with id fc5211831e89605a188727d42347e904e48246328fc0f341550390e88b4497ac
Nov 02 14:44:06.314950 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:06.314916    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt" event={"ID":"cc101b13-4831-47cc-ae27-f6533d5b360c","Type":"ContainerStarted","Data":"5569210bfc462a9a6229cf0b36372c596beeff057e61b85d8b0eb5cb9a028e7c"}
Nov 02 14:44:06.315411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:06.314970    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt" event={"ID":"cc101b13-4831-47cc-ae27-f6533d5b360c","Type":"ContainerStarted","Data":"fc5211831e89605a188727d42347e904e48246328fc0f341550390e88b4497ac"}
Nov 02 14:44:06.353589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:06.353537    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt" podStartSLOduration=1.353522775 podStartE2EDuration="1.353522775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:44:06.351187356 +0000 UTC m=+2810.894902949" watchObservedRunningTime="2026-11-02 14:44:06.353522775 +0000 UTC m=+2810.897238368"
Nov 02 14:44:07.322056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:07.322022    4492 generic.go:356] "Generic (PLEG): container finished" podID="cc101b13-4831-47cc-ae27-f6533d5b360c" containerID="5569210bfc462a9a6229cf0b36372c596beeff057e61b85d8b0eb5cb9a028e7c" exitCode=0
Nov 02 14:44:07.322523 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:07.322110    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt" event={"ID":"cc101b13-4831-47cc-ae27-f6533d5b360c","Type":"ContainerDied","Data":"5569210bfc462a9a6229cf0b36372c596beeff057e61b85d8b0eb5cb9a028e7c"}
Nov 02 14:44:08.472589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.472565    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:08.573940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.573899    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca\") pod \"cc101b13-4831-47cc-ae27-f6533d5b360c\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") "
Nov 02 14:44:08.574149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.574073    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca\") pod \"cc101b13-4831-47cc-ae27-f6533d5b360c\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") "
Nov 02 14:44:08.574149 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.574116    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf\" (UniqueName: \"kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf\") pod \"cc101b13-4831-47cc-ae27-f6533d5b360c\" (UID: \"cc101b13-4831-47cc-ae27-f6533d5b360c\") "
Nov 02 14:44:08.574319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.574291    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca" pod "cc101b13-4831-47cc-ae27-f6533d5b360c" (UID: "cc101b13-4831-47cc-ae27-f6533d5b360c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:08.574577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.574552    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:08.576300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.576271    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca" pod "cc101b13-4831-47cc-ae27-f6533d5b360c" (UID: "cc101b13-4831-47cc-ae27-f6533d5b360c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:08.576703 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.576686    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf" pod "cc101b13-4831-47cc-ae27-f6533d5b360c" (UID: "cc101b13-4831-47cc-ae27-f6533d5b360c"). InnerVolumeSpecName "kube-api-access-fcxhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:08.675105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.675060    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc101b13-4831-47cc-ae27-f6533d5b360c-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:08.675105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:08.675096    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxhf\" (UniqueName: \"kubernetes.io/projected/cc101b13-4831-47cc-ae27-f6533d5b360c-kube-api-access-fcxhf\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:09.336084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:09.336051    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"
Nov 02 14:44:09.336276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:09.336079    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt" event={"ID":"cc101b13-4831-47cc-ae27-f6533d5b360c","Type":"ContainerDied","Data":"fc5211831e89605a188727d42347e904e48246328fc0f341550390e88b4497ac"}
Nov 02 14:44:09.336276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:09.336138    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5211831e89605a188727d42347e904e48246328fc0f341550390e88b4497ac"
Nov 02 14:44:10.573533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:10.573503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:16.517094 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:44:16.517049    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:44:17.018026 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.017995    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:17.158798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.158761    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-jdk62"]
Nov 02 14:44:17.158973 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.158947    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62" podUID="96e0f510-4f38-4a33-9efb-5c18b53375b9" containerName="manager" containerID="cri-o://8afd635d58fa8bf0d402bdba4db7fd1dcd1aa6f6b2dd56d16103ce871faf38ea" gracePeriod=30
Nov 02 14:44:17.380444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.380416    4492 generic.go:356] "Generic (PLEG): container finished" podID="96e0f510-4f38-4a33-9efb-5c18b53375b9" containerID="8afd635d58fa8bf0d402bdba4db7fd1dcd1aa6f6b2dd56d16103ce871faf38ea" exitCode=0
Nov 02 14:44:17.380602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.380490    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62" event={"ID":"96e0f510-4f38-4a33-9efb-5c18b53375b9","Type":"ContainerDied","Data":"8afd635d58fa8bf0d402bdba4db7fd1dcd1aa6f6b2dd56d16103ce871faf38ea"}
Nov 02 14:44:17.411300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.411273    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:44:17.565469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.565347    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs\") pod \"96e0f510-4f38-4a33-9efb-5c18b53375b9\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") "
Nov 02 14:44:17.565865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.565498    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts\" (UniqueName: \"kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts\") pod \"96e0f510-4f38-4a33-9efb-5c18b53375b9\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") "
Nov 02 14:44:17.565865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.565537    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert\") pod \"96e0f510-4f38-4a33-9efb-5c18b53375b9\" (UID: \"96e0f510-4f38-4a33-9efb-5c18b53375b9\") "
Nov 02 14:44:17.567642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.567618    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs" pod "96e0f510-4f38-4a33-9efb-5c18b53375b9" (UID: "96e0f510-4f38-4a33-9efb-5c18b53375b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:17.567764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.567740    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert" pod "96e0f510-4f38-4a33-9efb-5c18b53375b9" (UID: "96e0f510-4f38-4a33-9efb-5c18b53375b9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:17.567816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.567760    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts" pod "96e0f510-4f38-4a33-9efb-5c18b53375b9" (UID: "96e0f510-4f38-4a33-9efb-5c18b53375b9"). InnerVolumeSpecName "kube-api-access-544ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:17.667420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.667338    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544ts\" (UniqueName: \"kubernetes.io/projected/96e0f510-4f38-4a33-9efb-5c18b53375b9-kube-api-access-544ts\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:17.667420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.667405    4492 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-webhook-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:17.667420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:17.667414    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e0f510-4f38-4a33-9efb-5c18b53375b9-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.387084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:18.387052    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62"
Nov 02 14:44:18.387084 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:18.387066    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-jdk62" event={"ID":"96e0f510-4f38-4a33-9efb-5c18b53375b9","Type":"ContainerDied","Data":"f3684b44be4d8489e189a50d9b04fc4af59263e12789b86ae893b07645659852"}
Nov 02 14:44:18.387310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:18.387147    4492 scope.go:122] "RemoveContainer" containerID="8afd635d58fa8bf0d402bdba4db7fd1dcd1aa6f6b2dd56d16103ce871faf38ea"
Nov 02 14:44:18.433873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:18.433836    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-jdk62"]
Nov 02 14:44:18.443323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:18.443293    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-jdk62"]
Nov 02 14:44:19.576725 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.576689    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:19.579899 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.579876    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e0f510-4f38-4a33-9efb-5c18b53375b9" path="/var/lib/kubelet/pods/96e0f510-4f38-4a33-9efb-5c18b53375b9/volumes"
Nov 02 14:44:19.712886 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.712845    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-zlxn4"]
Nov 02 14:44:19.713257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.713243    4492 cpu_manager.go:414] "containerMap: removing container" podUID="cc101b13-4831-47cc-ae27-f6533d5b360c" containerName="revoke-keys"
Nov 02 14:44:19.713257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.713257    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc101b13-4831-47cc-ae27-f6533d5b360c" containerName="revoke-keys"
Nov 02 14:44:19.713420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.713279    4492 cpu_manager.go:414] "containerMap: removing container" podUID="96e0f510-4f38-4a33-9efb-5c18b53375b9" containerName="manager"
Nov 02 14:44:19.713420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.713284    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e0f510-4f38-4a33-9efb-5c18b53375b9" containerName="manager"
Nov 02 14:44:19.713420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.713385    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc101b13-4831-47cc-ae27-f6533d5b360c" containerName="revoke-keys"
Nov 02 14:44:19.713420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.713397    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e0f510-4f38-4a33-9efb-5c18b53375b9" containerName="manager"
Nov 02 14:44:19.720153 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.720124    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.743893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.743858    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 14:44:19.744236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.744206    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 14:44:19.744391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.744315    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftwbw\""
Nov 02 14:44:19.744391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.744329    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 14:44:19.746589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.746559    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-zlxn4"]
Nov 02 14:44:19.890926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.890837    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/612257d4-87df-4fde-b706-29a34b2c62c8-webhook-cert\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.890926 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.890894    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/612257d4-87df-4fde-b706-29a34b2c62c8-metrics-certs\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.891141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.891040    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7pg\" (UniqueName: \"kubernetes.io/projected/612257d4-87df-4fde-b706-29a34b2c62c8-kube-api-access-pf7pg\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.992663 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.992623    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7pg\" (UniqueName: \"kubernetes.io/projected/612257d4-87df-4fde-b706-29a34b2c62c8-kube-api-access-pf7pg\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.992842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.992695    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/612257d4-87df-4fde-b706-29a34b2c62c8-webhook-cert\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.992842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.992729    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/612257d4-87df-4fde-b706-29a34b2c62c8-metrics-certs\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.995393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.995324    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/612257d4-87df-4fde-b706-29a34b2c62c8-webhook-cert\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:19.995393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:19.995388    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/612257d4-87df-4fde-b706-29a34b2c62c8-metrics-certs\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:20.014727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:20.014695    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7pg\" (UniqueName: \"kubernetes.io/projected/612257d4-87df-4fde-b706-29a34b2c62c8-kube-api-access-pf7pg\") pod \"maas-controller-5db8dc9fd8-zlxn4\" (UID: \"612257d4-87df-4fde-b706-29a34b2c62c8\") " pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:20.033812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:20.033782    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:20.343217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:20.343183    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-zlxn4"]
Nov 02 14:44:20.346631 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:44:20.346601    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612257d4_87df_4fde_b706_29a34b2c62c8.slice/crio-c4b9b6f426b5eb0359d0d9df15e6fa426c074a5c7a02cb9eaa06ccb471f90c61 WatchSource:0}: Error finding container c4b9b6f426b5eb0359d0d9df15e6fa426c074a5c7a02cb9eaa06ccb471f90c61: Status 404 returned error can't find the container with id c4b9b6f426b5eb0359d0d9df15e6fa426c074a5c7a02cb9eaa06ccb471f90c61
Nov 02 14:44:20.400883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:20.400844    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4" event={"ID":"612257d4-87df-4fde-b706-29a34b2c62c8","Type":"ContainerStarted","Data":"c4b9b6f426b5eb0359d0d9df15e6fa426c074a5c7a02cb9eaa06ccb471f90c61"}
Nov 02 14:44:20.518474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:20.518434    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:21.407501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:21.407466    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4" event={"ID":"612257d4-87df-4fde-b706-29a34b2c62c8","Type":"ContainerStarted","Data":"dcbd8c402c4c0c90b1d5edf9631b87e10595ef4a2f31d84068045f467d156da6"}
Nov 02 14:44:21.407894 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:21.407523    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:21.449557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:21.449509    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4" podStartSLOduration=2.026964848 podStartE2EDuration="2.44949406s" totalImagesPullingTime="422.529212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:44:21.446747959 +0000 UTC m=+2825.990463551" watchObservedRunningTime="2026-11-02 14:44:21.44949406 +0000 UTC m=+2825.993209657"
Nov 02 14:44:32.415938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:32.415899    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5db8dc9fd8-zlxn4"
Nov 02 14:44:35.400655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:35.400622    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:35.529885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:35.529849    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:40.752280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:40.752249    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:45.567785 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:45.567754    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:44:46.516466 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:44:46.516429    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:44:55.323188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.323151    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"]
Nov 02 14:44:55.323704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.323350    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn" podUID="3918932b-be47-42ad-8685-9bc895afe0f3" containerName="manager" containerID="cri-o://e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d" gracePeriod=10
Nov 02 14:44:55.576939 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.576874    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:44:55.598991 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.598957    4492 generic.go:356] "Generic (PLEG): container finished" podID="3918932b-be47-42ad-8685-9bc895afe0f3" containerID="e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d" exitCode=0
Nov 02 14:44:55.599156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.598993    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"
Nov 02 14:44:55.599156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.599041    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn" event={"ID":"3918932b-be47-42ad-8685-9bc895afe0f3","Type":"ContainerDied","Data":"e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d"}
Nov 02 14:44:55.599156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.599112    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn" event={"ID":"3918932b-be47-42ad-8685-9bc895afe0f3","Type":"ContainerDied","Data":"d30dcf5cf7118a11e07d47aeddbc891428a6f5371c0bba9c180cc23d87d923a1"}
Nov 02 14:44:55.599156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.599135    4492 scope.go:122] "RemoveContainer" containerID="e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d"
Nov 02 14:44:55.614874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.614854    4492 scope.go:122] "RemoveContainer" containerID="e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d"
Nov 02 14:44:55.615151 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:44:55.615132    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d\": container with ID starting with e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d not found: ID does not exist" containerID="e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d"
Nov 02 14:44:55.615239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.615164    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d"} err="failed to get container status \"e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d\": rpc error: code = NotFound desc = could not find container \"e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d\": container with ID starting with e494d9f7ec35e930e0803065a861044b55c6dc427534bcc53aa09f67f3c5718d not found: ID does not exist"
Nov 02 14:44:55.655875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.655842    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw\" (UniqueName: \"kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw\") pod \"3918932b-be47-42ad-8685-9bc895afe0f3\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") "
Nov 02 14:44:55.656083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.656066    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume\") pod \"3918932b-be47-42ad-8685-9bc895afe0f3\" (UID: \"3918932b-be47-42ad-8685-9bc895afe0f3\") "
Nov 02 14:44:55.656739 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.656488    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume" pod "3918932b-be47-42ad-8685-9bc895afe0f3" (UID: "3918932b-be47-42ad-8685-9bc895afe0f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:55.658228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.658200    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw" pod "3918932b-be47-42ad-8685-9bc895afe0f3" (UID: "3918932b-be47-42ad-8685-9bc895afe0f3"). InnerVolumeSpecName "kube-api-access-jb2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:55.757852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.757810    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2tw\" (UniqueName: \"kubernetes.io/projected/3918932b-be47-42ad-8685-9bc895afe0f3-kube-api-access-jb2tw\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:55.757852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.757847    4492 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3918932b-be47-42ad-8685-9bc895afe0f3-extensions-socket-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:44:55.940891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.940855    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"]
Nov 02 14:44:55.950592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:55.950560    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4flwn"]
Nov 02 14:44:57.580249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:44:57.580203    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3918932b-be47-42ad-8685-9bc895afe0f3" path="/var/lib/kubelet/pods/3918932b-be47-42ad-8685-9bc895afe0f3/volumes"
Nov 02 14:45:00.154234 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.154198    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"]
Nov 02 14:45:00.154650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.154633    4492 cpu_manager.go:414] "containerMap: removing container" podUID="3918932b-be47-42ad-8685-9bc895afe0f3" containerName="manager"
Nov 02 14:45:00.154650 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.154646    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3918932b-be47-42ad-8685-9bc895afe0f3" containerName="manager"
Nov 02 14:45:00.154746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.154734    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="3918932b-be47-42ad-8685-9bc895afe0f3" containerName="manager"
Nov 02 14:45:00.159220 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.159202    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.178367 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.178323    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"]
Nov 02 14:45:00.184545 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.184526    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:45:00.184680 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.184578    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:45:00.198054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.198024    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:45:00.303673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.303634    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n542h\" (UniqueName: \"kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h\") pod \"maas-api-key-cleanup-29893845-cjcmn\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.303865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.303788    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893845-cjcmn\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.404930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.404839    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893845-cjcmn\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.404930 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.404897    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n542h\" (UniqueName: \"kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h\") pod \"maas-api-key-cleanup-29893845-cjcmn\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.405519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.405499    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893845-cjcmn\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.426692 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.426661    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n542h\" (UniqueName: \"kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h\") pod \"maas-api-key-cleanup-29893845-cjcmn\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.471510 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.471475    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:00.769543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:00.769500    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"]
Nov 02 14:45:00.773004 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:45:00.772972    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0684a323_c7f9_44ee_a050_5d5eadd7f298.slice/crio-1c2b361cf90588e2c50d586b7c70a9e0e2ef69ffb032a558ae8af9a05d6eaac8 WatchSource:0}: Error finding container 1c2b361cf90588e2c50d586b7c70a9e0e2ef69ffb032a558ae8af9a05d6eaac8: Status 404 returned error can't find the container with id 1c2b361cf90588e2c50d586b7c70a9e0e2ef69ffb032a558ae8af9a05d6eaac8
Nov 02 14:45:01.634761 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:01.634731    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" event={"ID":"0684a323-c7f9-44ee-a050-5d5eadd7f298","Type":"ContainerStarted","Data":"2ca55048f4c521c6d45e6086a7153fd6e07a80fd2406dee0f8e5c33caf400861"}
Nov 02 14:45:01.635232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:01.634783    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" event={"ID":"0684a323-c7f9-44ee-a050-5d5eadd7f298","Type":"ContainerStarted","Data":"1c2b361cf90588e2c50d586b7c70a9e0e2ef69ffb032a558ae8af9a05d6eaac8"}
Nov 02 14:45:01.672465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:01.672409    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" podStartSLOduration=1.672391529 podStartE2EDuration="1.672391529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:01.670471788 +0000 UTC m=+2866.214187394" watchObservedRunningTime="2026-11-02 14:45:01.672391529 +0000 UTC m=+2866.216107123"
Nov 02 14:45:02.641850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:02.641808    4492 generic.go:356] "Generic (PLEG): container finished" podID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerID="2ca55048f4c521c6d45e6086a7153fd6e07a80fd2406dee0f8e5c33caf400861" exitCode=60
Nov 02 14:45:02.642269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:02.641893    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" event={"ID":"0684a323-c7f9-44ee-a050-5d5eadd7f298","Type":"ContainerDied","Data":"2ca55048f4c521c6d45e6086a7153fd6e07a80fd2406dee0f8e5c33caf400861"}
Nov 02 14:45:02.642269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:02.642119    4492 scope.go:122] "RemoveContainer" containerID="2ca55048f4c521c6d45e6086a7153fd6e07a80fd2406dee0f8e5c33caf400861"
Nov 02 14:45:03.648652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:03.648611    4492 generic.go:356] "Generic (PLEG): container finished" podID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerID="a89436b22b34de9cf300e42c5febed377c080ae402afbfde1e4a2e8eee785a16" exitCode=60
Nov 02 14:45:03.649091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:03.648701    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" event={"ID":"0684a323-c7f9-44ee-a050-5d5eadd7f298","Type":"ContainerDied","Data":"a89436b22b34de9cf300e42c5febed377c080ae402afbfde1e4a2e8eee785a16"}
Nov 02 14:45:03.649091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:03.648762    4492 scope.go:122] "RemoveContainer" containerID="2ca55048f4c521c6d45e6086a7153fd6e07a80fd2406dee0f8e5c33caf400861"
Nov 02 14:45:03.649091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:03.648973    4492 scope.go:122] "RemoveContainer" containerID="a89436b22b34de9cf300e42c5febed377c080ae402afbfde1e4a2e8eee785a16"
Nov 02 14:45:03.649215 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:45:03.649198    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893845-cjcmn_odh-ai-gateway-infra(0684a323-c7f9-44ee-a050-5d5eadd7f298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298"
Nov 02 14:45:04.656195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:04.656166    4492 scope.go:122] "RemoveContainer" containerID="a89436b22b34de9cf300e42c5febed377c080ae402afbfde1e4a2e8eee785a16"
Nov 02 14:45:04.656665 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:45:04.656469    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893845-cjcmn_odh-ai-gateway-infra(0684a323-c7f9-44ee-a050-5d5eadd7f298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298"
Nov 02 14:45:10.718996 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:10.718957    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:45:15.577722 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:15.577694    4492 scope.go:122] "RemoveContainer" containerID="a89436b22b34de9cf300e42c5febed377c080ae402afbfde1e4a2e8eee785a16"
Nov 02 14:45:16.515406 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:45:16.515370    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:45:16.718316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:16.718280    4492 generic.go:356] "Generic (PLEG): container finished" podID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerID="134eb0f22d6bf8790762acb54f50dd799a952d9e4ca03d098436d7ed1454def4" exitCode=60
Nov 02 14:45:16.718782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:16.718378    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" event={"ID":"0684a323-c7f9-44ee-a050-5d5eadd7f298","Type":"ContainerDied","Data":"134eb0f22d6bf8790762acb54f50dd799a952d9e4ca03d098436d7ed1454def4"}
Nov 02 14:45:16.718782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:16.718438    4492 scope.go:122] "RemoveContainer" containerID="a89436b22b34de9cf300e42c5febed377c080ae402afbfde1e4a2e8eee785a16"
Nov 02 14:45:16.718782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:16.718741    4492 scope.go:122] "RemoveContainer" containerID="134eb0f22d6bf8790762acb54f50dd799a952d9e4ca03d098436d7ed1454def4"
Nov 02 14:45:16.719037 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:45:16.719015    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893845-cjcmn_odh-ai-gateway-infra(0684a323-c7f9-44ee-a050-5d5eadd7f298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298"
Nov 02 14:45:17.768061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:17.768029    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"]
Nov 02 14:45:17.905332 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:17.905306    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:17.978399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:17.978367    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca\") pod \"0684a323-c7f9-44ee-a050-5d5eadd7f298\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") "
Nov 02 14:45:17.978565 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:17.978453    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h\" (UniqueName: \"kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h\") pod \"0684a323-c7f9-44ee-a050-5d5eadd7f298\" (UID: \"0684a323-c7f9-44ee-a050-5d5eadd7f298\") "
Nov 02 14:45:17.978774 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:17.978743    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca" pod "0684a323-c7f9-44ee-a050-5d5eadd7f298" (UID: "0684a323-c7f9-44ee-a050-5d5eadd7f298"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:17.980631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:17.980606    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h" pod "0684a323-c7f9-44ee-a050-5d5eadd7f298" (UID: "0684a323-c7f9-44ee-a050-5d5eadd7f298"). InnerVolumeSpecName "kube-api-access-n542h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:18.080137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.080061    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684a323-c7f9-44ee-a050-5d5eadd7f298-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:45:18.080137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.080090    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n542h\" (UniqueName: \"kubernetes.io/projected/0684a323-c7f9-44ee-a050-5d5eadd7f298-kube-api-access-n542h\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:45:18.731438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.731411    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"
Nov 02 14:45:18.731627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.731429    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn" event={"ID":"0684a323-c7f9-44ee-a050-5d5eadd7f298","Type":"ContainerDied","Data":"1c2b361cf90588e2c50d586b7c70a9e0e2ef69ffb032a558ae8af9a05d6eaac8"}
Nov 02 14:45:18.731627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.731494    4492 scope.go:122] "RemoveContainer" containerID="134eb0f22d6bf8790762acb54f50dd799a952d9e4ca03d098436d7ed1454def4"
Nov 02 14:45:18.778449 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.778420    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"]
Nov 02 14:45:18.788093 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:18.788065    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cjcmn"]
Nov 02 14:45:19.579669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:19.579634    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" path="/var/lib/kubelet/pods/0684a323-c7f9-44ee-a050-5d5eadd7f298/volumes"
Nov 02 14:45:20.650450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:20.650414    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:45:40.857034 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:40.856998    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:45:46.516809 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:45:46.516772    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:45:55.062905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:55.062867    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:45:58.614162 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614075    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"]
Nov 02 14:45:58.614712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614691    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.614794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614716    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.614794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614736    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.614794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614745    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.614954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614870    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.614954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614884    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.614954 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.614898    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:45:58.617648 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.617626    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.640411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.640378    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85kp5\""
Nov 02 14:45:58.643237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.643214    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"]
Nov 02 14:45:58.701006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.700962    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42f5db3c-64e6-4d87-ab71-01fb74715791-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fgnlf\" (UID: \"42f5db3c-64e6-4d87-ab71-01fb74715791\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.701314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.701074    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbt5j\" (UniqueName: \"kubernetes.io/projected/42f5db3c-64e6-4d87-ab71-01fb74715791-kube-api-access-sbt5j\") pod \"kuadrant-operator-controller-manager-678d87766f-fgnlf\" (UID: \"42f5db3c-64e6-4d87-ab71-01fb74715791\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.802312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.802273    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42f5db3c-64e6-4d87-ab71-01fb74715791-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fgnlf\" (UID: \"42f5db3c-64e6-4d87-ab71-01fb74715791\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.802536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.802399    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbt5j\" (UniqueName: \"kubernetes.io/projected/42f5db3c-64e6-4d87-ab71-01fb74715791-kube-api-access-sbt5j\") pod \"kuadrant-operator-controller-manager-678d87766f-fgnlf\" (UID: \"42f5db3c-64e6-4d87-ab71-01fb74715791\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.802777 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.802754    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42f5db3c-64e6-4d87-ab71-01fb74715791-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fgnlf\" (UID: \"42f5db3c-64e6-4d87-ab71-01fb74715791\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.826714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.826680    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbt5j\" (UniqueName: \"kubernetes.io/projected/42f5db3c-64e6-4d87-ab71-01fb74715791-kube-api-access-sbt5j\") pod \"kuadrant-operator-controller-manager-678d87766f-fgnlf\" (UID: \"42f5db3c-64e6-4d87-ab71-01fb74715791\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:58.930674 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:58.930644    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:45:59.263862 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:59.263767    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"]
Nov 02 14:45:59.267466 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:45:59.267437    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f5db3c_64e6_4d87_ab71_01fb74715791.slice/crio-6401b4aff67f68749b49a3dee31ce6be9e7cf282c8126d73e39660be0a911650 WatchSource:0}: Error finding container 6401b4aff67f68749b49a3dee31ce6be9e7cf282c8126d73e39660be0a911650: Status 404 returned error can't find the container with id 6401b4aff67f68749b49a3dee31ce6be9e7cf282c8126d73e39660be0a911650
Nov 02 14:45:59.953015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:59.952979    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf" event={"ID":"42f5db3c-64e6-4d87-ab71-01fb74715791","Type":"ContainerStarted","Data":"39c73f70f74af3d766ba85d7dd262c437368b4d5678875c16a81d59d400453d6"}
Nov 02 14:45:59.953453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:59.953035    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf" event={"ID":"42f5db3c-64e6-4d87-ab71-01fb74715791","Type":"ContainerStarted","Data":"6401b4aff67f68749b49a3dee31ce6be9e7cf282c8126d73e39660be0a911650"}
Nov 02 14:45:59.953453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:45:59.953182    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:46:00.013126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:00.013069    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf" podStartSLOduration=2.013053293 podStartE2EDuration="2.013053293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:46:00.002601823 +0000 UTC m=+2924.546317417" watchObservedRunningTime="2026-11-02 14:46:00.013053293 +0000 UTC m=+2924.556768884"
Nov 02 14:46:10.853221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:10.853183    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:46:10.960945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:10.960919    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fgnlf"
Nov 02 14:46:16.516562 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:46:16.516522    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:46:21.986500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:21.986458    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:46:21.987383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:21.987111    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:46:30.773236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:30.773201    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:46:31.641714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.641676    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"]
Nov 02 14:46:31.642130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.642117    4492 cpu_manager.go:414] "containerMap: removing container" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:46:31.642177 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.642132    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0684a323-c7f9-44ee-a050-5d5eadd7f298" containerName="cleanup"
Nov 02 14:46:31.646639 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.646622    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.663488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.663460    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"]
Nov 02 14:46:31.670347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.670321    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:46:31.670586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.670321    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:46:31.670586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.670393    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:46:31.670781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.670430    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:46:31.786009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.785967    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.786515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.786148    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.786515 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.786253    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbsp\" (UniqueName: \"kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.887082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.887042    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbsp\" (UniqueName: \"kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.887282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.887120    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.887282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.887197    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.887952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.887930    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.889969 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.889951    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.909317 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.909246    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbsp\" (UniqueName: \"kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp\") pod \"maas-api-revoke-sub-d576e68b-tvvsz\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:31.960048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:31.960008    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:32.270283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:32.270247    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"]
Nov 02 14:46:32.271866 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:46:32.271840    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b5dab7_0378_47c8_83cf_8328136f7b4e.slice/crio-624711655bf36594a78719f4016b981c746df97988bfb754b3e789ffaa529dc5 WatchSource:0}: Error finding container 624711655bf36594a78719f4016b981c746df97988bfb754b3e789ffaa529dc5: Status 404 returned error can't find the container with id 624711655bf36594a78719f4016b981c746df97988bfb754b3e789ffaa529dc5
Nov 02 14:46:33.170876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:33.170842    4492 generic.go:356] "Generic (PLEG): container finished" podID="00b5dab7-0378-47c8-83cf-8328136f7b4e" containerID="71e643a6ac50d84544db27fb0f86413fa2b49a630e3d300357bc3f1482608d78" exitCode=0
Nov 02 14:46:33.171253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:33.170915    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz" event={"ID":"00b5dab7-0378-47c8-83cf-8328136f7b4e","Type":"ContainerDied","Data":"71e643a6ac50d84544db27fb0f86413fa2b49a630e3d300357bc3f1482608d78"}
Nov 02 14:46:33.171253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:33.170951    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz" event={"ID":"00b5dab7-0378-47c8-83cf-8328136f7b4e","Type":"ContainerStarted","Data":"624711655bf36594a78719f4016b981c746df97988bfb754b3e789ffaa529dc5"}
Nov 02 14:46:34.313865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.313829    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:34.412906 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.412874    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp\" (UniqueName: \"kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp\") pod \"00b5dab7-0378-47c8-83cf-8328136f7b4e\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") "
Nov 02 14:46:34.413088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.412961    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca\") pod \"00b5dab7-0378-47c8-83cf-8328136f7b4e\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") "
Nov 02 14:46:34.413088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.412998    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca\") pod \"00b5dab7-0378-47c8-83cf-8328136f7b4e\" (UID: \"00b5dab7-0378-47c8-83cf-8328136f7b4e\") "
Nov 02 14:46:34.413633 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.413584    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca" pod "00b5dab7-0378-47c8-83cf-8328136f7b4e" (UID: "00b5dab7-0378-47c8-83cf-8328136f7b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:34.415139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.415116    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca" pod "00b5dab7-0378-47c8-83cf-8328136f7b4e" (UID: "00b5dab7-0378-47c8-83cf-8328136f7b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:34.415282 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.415266    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp" pod "00b5dab7-0378-47c8-83cf-8328136f7b4e" (UID: "00b5dab7-0378-47c8-83cf-8328136f7b4e"). InnerVolumeSpecName "kube-api-access-jhbsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:34.514539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.514444    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbsp\" (UniqueName: \"kubernetes.io/projected/00b5dab7-0378-47c8-83cf-8328136f7b4e-kube-api-access-jhbsp\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:46:34.514539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.514479    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:46:34.514539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:34.514488    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b5dab7-0378-47c8-83cf-8328136f7b4e-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:46:35.181865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:35.181826    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"
Nov 02 14:46:35.182051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:35.181866    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz" event={"ID":"00b5dab7-0378-47c8-83cf-8328136f7b4e","Type":"ContainerDied","Data":"624711655bf36594a78719f4016b981c746df97988bfb754b3e789ffaa529dc5"}
Nov 02 14:46:35.182051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:35.181914    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="624711655bf36594a78719f4016b981c746df97988bfb754b3e789ffaa529dc5"
Nov 02 14:46:41.004119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:46:41.004084    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:46:46.531977 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:46:46.531924    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:47:01.302137 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:01.302099    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:01.420308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:01.420271    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:04.951519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:04.951478    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:47:04.952176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:04.952143    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:47:04.973496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:04.973461    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:47:04.973771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:04.973742    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:47:05.042380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:05.041489    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:47:05.042380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:05.041592    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:47:07.915635 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.915590    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"]
Nov 02 14:47:07.916260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.916238    4492 cpu_manager.go:414] "containerMap: removing container" podUID="00b5dab7-0378-47c8-83cf-8328136f7b4e" containerName="revoke-keys"
Nov 02 14:47:07.916398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.916265    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b5dab7-0378-47c8-83cf-8328136f7b4e" containerName="revoke-keys"
Nov 02 14:47:07.916398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.916380    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b5dab7-0378-47c8-83cf-8328136f7b4e" containerName="revoke-keys"
Nov 02 14:47:07.919811 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.919790    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:07.938585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.938549    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"]
Nov 02 14:47:07.944985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.944957    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:07.944985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.944973    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:07.945203 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.945022    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:07.948788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:07.948768    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:47:08.027215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.027178    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.027435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.027233    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv77v\" (UniqueName: \"kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.027435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.027326    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.128719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.128680    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.128914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.128734    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv77v\" (UniqueName: \"kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.128914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.128760    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.129505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.129484    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.131363 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.131341    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.155014 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.154986    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv77v\" (UniqueName: \"kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v\") pod \"maas-api-revoke-sub-c5af728b-65n8l\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.233879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.233831    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:08.538205 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:08.538099    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"]
Nov 02 14:47:08.541011 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:47:08.540981    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc414e5_ae0b_4b09_8613_d0bba4791181.slice/crio-6bba4e3e99f1b20f1befd513a0345663ffddb64629d9bed8017f3a9798163609 WatchSource:0}: Error finding container 6bba4e3e99f1b20f1befd513a0345663ffddb64629d9bed8017f3a9798163609: Status 404 returned error can't find the container with id 6bba4e3e99f1b20f1befd513a0345663ffddb64629d9bed8017f3a9798163609
Nov 02 14:47:09.501760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:09.501725    4492 generic.go:356] "Generic (PLEG): container finished" podID="2fc414e5-ae0b-4b09-8613-d0bba4791181" containerID="5ec3f59559db8b417e966f144e795f22678700b31bac420a60b5899c33acf190" exitCode=0
Nov 02 14:47:09.502181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:09.501815    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l" event={"ID":"2fc414e5-ae0b-4b09-8613-d0bba4791181","Type":"ContainerDied","Data":"5ec3f59559db8b417e966f144e795f22678700b31bac420a60b5899c33acf190"}
Nov 02 14:47:09.502181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:09.501885    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l" event={"ID":"2fc414e5-ae0b-4b09-8613-d0bba4791181","Type":"ContainerStarted","Data":"6bba4e3e99f1b20f1befd513a0345663ffddb64629d9bed8017f3a9798163609"}
Nov 02 14:47:10.655273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.655241    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:10.758798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.758701    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca\") pod \"2fc414e5-ae0b-4b09-8613-d0bba4791181\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") "
Nov 02 14:47:10.758798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.758748    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v\" (UniqueName: \"kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v\") pod \"2fc414e5-ae0b-4b09-8613-d0bba4791181\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") "
Nov 02 14:47:10.758798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.758775    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca\") pod \"2fc414e5-ae0b-4b09-8613-d0bba4791181\" (UID: \"2fc414e5-ae0b-4b09-8613-d0bba4791181\") "
Nov 02 14:47:10.759217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.759194    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca" pod "2fc414e5-ae0b-4b09-8613-d0bba4791181" (UID: "2fc414e5-ae0b-4b09-8613-d0bba4791181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:10.761106 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.761085    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v" pod "2fc414e5-ae0b-4b09-8613-d0bba4791181" (UID: "2fc414e5-ae0b-4b09-8613-d0bba4791181"). InnerVolumeSpecName "kube-api-access-nv77v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:10.761193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.761141    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca" pod "2fc414e5-ae0b-4b09-8613-d0bba4791181" (UID: "2fc414e5-ae0b-4b09-8613-d0bba4791181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:10.860058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.860024    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:10.860058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.860056    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv77v\" (UniqueName: \"kubernetes.io/projected/2fc414e5-ae0b-4b09-8613-d0bba4791181-kube-api-access-nv77v\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:10.860058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.860066    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc414e5-ae0b-4b09-8613-d0bba4791181-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:10.952602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:10.952573    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:11.514016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:11.513987    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"
Nov 02 14:47:11.514262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:11.513983    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l" event={"ID":"2fc414e5-ae0b-4b09-8613-d0bba4791181","Type":"ContainerDied","Data":"6bba4e3e99f1b20f1befd513a0345663ffddb64629d9bed8017f3a9798163609"}
Nov 02 14:47:11.514262 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:11.514147    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bba4e3e99f1b20f1befd513a0345663ffddb64629d9bed8017f3a9798163609"
Nov 02 14:47:16.516696 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:47:16.516658    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:47:28.298507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.294142    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"]
Nov 02 14:47:28.298507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.295041    4492 cpu_manager.go:414] "containerMap: removing container" podUID="2fc414e5-ae0b-4b09-8613-d0bba4791181" containerName="revoke-keys"
Nov 02 14:47:28.298507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.295060    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc414e5-ae0b-4b09-8613-d0bba4791181" containerName="revoke-keys"
Nov 02 14:47:28.298507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.295273    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc414e5-ae0b-4b09-8613-d0bba4791181" containerName="revoke-keys"
Nov 02 14:47:28.299250 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.299231    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.315493 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.315466    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"]
Nov 02 14:47:28.321264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.321236    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:28.321431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.321243    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:28.321431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.321290    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:28.321996 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.321977    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:47:28.348170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.348130    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9t5c\" (UniqueName: \"kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.348499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.348471    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.348608 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.348532    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.449855 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.449811    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.450089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.449867    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.450089 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.449982    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9t5c\" (UniqueName: \"kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.450534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.450504    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.452470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.452445    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.486119 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.486081    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9t5c\" (UniqueName: \"kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c\") pod \"maas-api-revoke-sub-6ac1e13d-5j9pt\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.613569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.613489    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:28.927433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:28.927400    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"]
Nov 02 14:47:28.930780 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:47:28.930749    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb778013_1129_4e7d_a691_33fe01b7ad16.slice/crio-031427dd40ea6c1fe0406d618e97a6683f1dc08c6e824ed05e638e6eef6ce1d6 WatchSource:0}: Error finding container 031427dd40ea6c1fe0406d618e97a6683f1dc08c6e824ed05e638e6eef6ce1d6: Status 404 returned error can't find the container with id 031427dd40ea6c1fe0406d618e97a6683f1dc08c6e824ed05e638e6eef6ce1d6
Nov 02 14:47:29.618427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:29.618310    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt" event={"ID":"eb778013-1129-4e7d-a691-33fe01b7ad16","Type":"ContainerStarted","Data":"77821bc25e54c8b0bacd927d24c3a4ff38e595bd34f97c7e63e3fa6363b5a348"}
Nov 02 14:47:29.618427 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:29.618385    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt" event={"ID":"eb778013-1129-4e7d-a691-33fe01b7ad16","Type":"ContainerStarted","Data":"031427dd40ea6c1fe0406d618e97a6683f1dc08c6e824ed05e638e6eef6ce1d6"}
Nov 02 14:47:29.656207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:29.656160    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt" podStartSLOduration=1.656145189 podStartE2EDuration="1.656145189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:29.655223296 +0000 UTC m=+3014.198938891" watchObservedRunningTime="2026-11-02 14:47:29.656145189 +0000 UTC m=+3014.199860782"
Nov 02 14:47:30.625295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:30.625252    4492 generic.go:356] "Generic (PLEG): container finished" podID="eb778013-1129-4e7d-a691-33fe01b7ad16" containerID="77821bc25e54c8b0bacd927d24c3a4ff38e595bd34f97c7e63e3fa6363b5a348" exitCode=0
Nov 02 14:47:30.625295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:30.625294    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt" event={"ID":"eb778013-1129-4e7d-a691-33fe01b7ad16","Type":"ContainerDied","Data":"77821bc25e54c8b0bacd927d24c3a4ff38e595bd34f97c7e63e3fa6363b5a348"}
Nov 02 14:47:30.907407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:30.907289    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:31.770889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.770862    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:31.886187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.886091    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c\" (UniqueName: \"kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c\") pod \"eb778013-1129-4e7d-a691-33fe01b7ad16\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") "
Nov 02 14:47:31.886187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.886148    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca\") pod \"eb778013-1129-4e7d-a691-33fe01b7ad16\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") "
Nov 02 14:47:31.886187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.886176    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca\") pod \"eb778013-1129-4e7d-a691-33fe01b7ad16\" (UID: \"eb778013-1129-4e7d-a691-33fe01b7ad16\") "
Nov 02 14:47:31.886647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.886618    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca" pod "eb778013-1129-4e7d-a691-33fe01b7ad16" (UID: "eb778013-1129-4e7d-a691-33fe01b7ad16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:31.888433 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.888406    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca" pod "eb778013-1129-4e7d-a691-33fe01b7ad16" (UID: "eb778013-1129-4e7d-a691-33fe01b7ad16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:31.888601 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.888587    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c" pod "eb778013-1129-4e7d-a691-33fe01b7ad16" (UID: "eb778013-1129-4e7d-a691-33fe01b7ad16"). InnerVolumeSpecName "kube-api-access-j9t5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:31.987905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.987865    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:31.987905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.987895    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb778013-1129-4e7d-a691-33fe01b7ad16-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:31.987905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:31.987905    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9t5c\" (UniqueName: \"kubernetes.io/projected/eb778013-1129-4e7d-a691-33fe01b7ad16-kube-api-access-j9t5c\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:32.637704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:32.637673    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"
Nov 02 14:47:32.637980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:32.637709    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt" event={"ID":"eb778013-1129-4e7d-a691-33fe01b7ad16","Type":"ContainerDied","Data":"031427dd40ea6c1fe0406d618e97a6683f1dc08c6e824ed05e638e6eef6ce1d6"}
Nov 02 14:47:32.637980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:32.637754    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="031427dd40ea6c1fe0406d618e97a6683f1dc08c6e824ed05e638e6eef6ce1d6"
Nov 02 14:47:38.729126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.729091    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"]
Nov 02 14:47:38.729543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.729528    4492 cpu_manager.go:414] "containerMap: removing container" podUID="eb778013-1129-4e7d-a691-33fe01b7ad16" containerName="revoke-keys"
Nov 02 14:47:38.729543 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.729542    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb778013-1129-4e7d-a691-33fe01b7ad16" containerName="revoke-keys"
Nov 02 14:47:38.729630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.729619    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb778013-1129-4e7d-a691-33fe01b7ad16" containerName="revoke-keys"
Nov 02 14:47:38.733060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.733033    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.747174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.747141    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:38.757118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.757089    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:47:38.757118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.757114    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:38.757407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.757185    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:38.761771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.761749    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"]
Nov 02 14:47:38.863440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.863388    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.863640 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.863504    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbg5\" (UniqueName: \"kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.863640 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.863575    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.964863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.964831    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbg5\" (UniqueName: \"kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.965057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.964898    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.965057 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.965024    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.965728 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.965698    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.967589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.967570    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:38.988187 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:38.988127    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbg5\" (UniqueName: \"kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5\") pod \"maas-api-revoke-sub-7b123093-n5gd2\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:39.053383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:39.053319    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:39.352260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:39.352162    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"]
Nov 02 14:47:39.356414 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:47:39.356388    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode650c7f2_3d46_4aad_b1ba_bdb51b4a60b3.slice/crio-19ac2c9f63652a1b3a6e6f42834adf0b146ed28e4cf835800babb9d3ff933154 WatchSource:0}: Error finding container 19ac2c9f63652a1b3a6e6f42834adf0b146ed28e4cf835800babb9d3ff933154: Status 404 returned error can't find the container with id 19ac2c9f63652a1b3a6e6f42834adf0b146ed28e4cf835800babb9d3ff933154
Nov 02 14:47:39.684696 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:39.684657    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2" event={"ID":"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3","Type":"ContainerStarted","Data":"f6a60e9dd8f392b8bb24f1542cfba9af26bb7706c3747186973604a2bf5bdd7e"}
Nov 02 14:47:39.684864 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:39.684716    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2" event={"ID":"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3","Type":"ContainerStarted","Data":"19ac2c9f63652a1b3a6e6f42834adf0b146ed28e4cf835800babb9d3ff933154"}
Nov 02 14:47:39.722567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:39.722509    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2" podStartSLOduration=1.722494408 podStartE2EDuration="1.722494408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:39.721069165 +0000 UTC m=+3024.264784760" watchObservedRunningTime="2026-11-02 14:47:39.722494408 +0000 UTC m=+3024.266210042"
Nov 02 14:47:40.714462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:40.714420    4492 generic.go:356] "Generic (PLEG): container finished" podID="e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" containerID="f6a60e9dd8f392b8bb24f1542cfba9af26bb7706c3747186973604a2bf5bdd7e" exitCode=0
Nov 02 14:47:40.714992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:40.714508    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2" event={"ID":"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3","Type":"ContainerDied","Data":"f6a60e9dd8f392b8bb24f1542cfba9af26bb7706c3747186973604a2bf5bdd7e"}
Nov 02 14:47:41.161054 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:41.160864    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:42.511885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.511544    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:42.606007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.605960    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca\") pod \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") "
Nov 02 14:47:42.607444 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.607405    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca\") pod \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") "
Nov 02 14:47:42.607994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.607975    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5\" (UniqueName: \"kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5\") pod \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\" (UID: \"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3\") "
Nov 02 14:47:42.608808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.607304    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca" pod "e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" (UID: "e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:42.630668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.630606    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5" pod "e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" (UID: "e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3"). InnerVolumeSpecName "kube-api-access-spbg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:42.631315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.631278    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca" pod "e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" (UID: "e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:42.713450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.713137    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:42.713450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.713180    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:42.713450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.713198    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbg5\" (UniqueName: \"kubernetes.io/projected/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3-kube-api-access-spbg5\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:42.829795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.827678    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2" event={"ID":"e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3","Type":"ContainerDied","Data":"19ac2c9f63652a1b3a6e6f42834adf0b146ed28e4cf835800babb9d3ff933154"}
Nov 02 14:47:42.829795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.827746    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ac2c9f63652a1b3a6e6f42834adf0b146ed28e4cf835800babb9d3ff933154"
Nov 02 14:47:42.829795 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:42.827822    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"
Nov 02 14:47:44.968195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:44.968163    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:46.097369 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:46.097321    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:47:46.516864 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:47:46.516827    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:47:49.179868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.179831    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"]
Nov 02 14:47:49.180878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.180829    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" containerName="revoke-keys"
Nov 02 14:47:49.180878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.180854    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" containerName="revoke-keys"
Nov 02 14:47:49.181075 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.180948    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" containerName="revoke-keys"
Nov 02 14:47:49.184667 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.184647    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.201031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.201008    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"]
Nov 02 14:47:49.221398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.206892    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:49.221398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.206904    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:49.221398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.207584    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:47:49.221398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.211481    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:49.296456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.296418    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.296615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.296495    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.296682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.296642    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pj2\" (UniqueName: \"kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.397922 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.397879    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.398114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.397986    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pj2\" (UniqueName: \"kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.398114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.398025    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.398831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.398804    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.400459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.400433    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.419624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.419592    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pj2\" (UniqueName: \"kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2\") pod \"maas-api-revoke-sub-bd3b9634-ldx5r\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.504852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.504762    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:49.807566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.807468    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"]
Nov 02 14:47:49.811145 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:47:49.811113    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22ddf63_6490_43c0_ac6e_519ab0adcf69.slice/crio-331c0d168ccdfe7be0d8154cde61b216b23ce1e9ee180ba2c8045c69c525a332 WatchSource:0}: Error finding container 331c0d168ccdfe7be0d8154cde61b216b23ce1e9ee180ba2c8045c69c525a332: Status 404 returned error can't find the container with id 331c0d168ccdfe7be0d8154cde61b216b23ce1e9ee180ba2c8045c69c525a332
Nov 02 14:47:49.868479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:49.868441    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r" event={"ID":"a22ddf63-6490-43c0-ac6e-519ab0adcf69","Type":"ContainerStarted","Data":"331c0d168ccdfe7be0d8154cde61b216b23ce1e9ee180ba2c8045c69c525a332"}
Nov 02 14:47:50.881396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:50.880300    4492 generic.go:356] "Generic (PLEG): container finished" podID="a22ddf63-6490-43c0-ac6e-519ab0adcf69" containerID="6464ba3e42c3f7fdedc3a676f482ebcaae99b3793db8b3b4025f2eea832cd576" exitCode=0
Nov 02 14:47:50.881396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:50.880427    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r" event={"ID":"a22ddf63-6490-43c0-ac6e-519ab0adcf69","Type":"ContainerDied","Data":"6464ba3e42c3f7fdedc3a676f482ebcaae99b3793db8b3b4025f2eea832cd576"}
Nov 02 14:47:52.605391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.604531    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:52.723381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.687251    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2\" (UniqueName: \"kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2\") pod \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") "
Nov 02 14:47:52.723381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.698098    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca\") pod \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") "
Nov 02 14:47:52.723381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.702636    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca" pod "a22ddf63-6490-43c0-ac6e-519ab0adcf69" (UID: "a22ddf63-6490-43c0-ac6e-519ab0adcf69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:52.723381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.707433    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca\") pod \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\" (UID: \"a22ddf63-6490-43c0-ac6e-519ab0adcf69\") "
Nov 02 14:47:52.734434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.731614    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:52.985380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.985229    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r" event={"ID":"a22ddf63-6490-43c0-ac6e-519ab0adcf69","Type":"ContainerDied","Data":"331c0d168ccdfe7be0d8154cde61b216b23ce1e9ee180ba2c8045c69c525a332"}
Nov 02 14:47:52.985380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.985296    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331c0d168ccdfe7be0d8154cde61b216b23ce1e9ee180ba2c8045c69c525a332"
Nov 02 14:47:52.985671 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.985428    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"
Nov 02 14:47:52.989377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.986333    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2" pod "a22ddf63-6490-43c0-ac6e-519ab0adcf69" (UID: "a22ddf63-6490-43c0-ac6e-519ab0adcf69"). InnerVolumeSpecName "kube-api-access-j7pj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:52.989377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:52.986440    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca" pod "a22ddf63-6490-43c0-ac6e-519ab0adcf69" (UID: "a22ddf63-6490-43c0-ac6e-519ab0adcf69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:53.072381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:53.072120    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7pj2\" (UniqueName: \"kubernetes.io/projected/a22ddf63-6490-43c0-ac6e-519ab0adcf69-kube-api-access-j7pj2\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:53.072381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:53.072168    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a22ddf63-6490-43c0-ac6e-519ab0adcf69-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:47:59.652222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.652186    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"]
Nov 02 14:47:59.652831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.652662    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a22ddf63-6490-43c0-ac6e-519ab0adcf69" containerName="revoke-keys"
Nov 02 14:47:59.652831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.652680    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22ddf63-6490-43c0-ac6e-519ab0adcf69" containerName="revoke-keys"
Nov 02 14:47:59.652831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.652772    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22ddf63-6490-43c0-ac6e-519ab0adcf69" containerName="revoke-keys"
Nov 02 14:47:59.655965 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.655943    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.671399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.671319    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:59.671535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.671455    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:47:59.671615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.671546    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:59.686232 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.686206    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:59.690682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.690646    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"]
Nov 02 14:47:59.747189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.747144    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.747391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.747209    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.747391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.747298    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhhd\" (UniqueName: \"kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.848670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.848635    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.848860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.848688    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.848860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.848718    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhhd\" (UniqueName: \"kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.849461 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.849373    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.851229 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.851205    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.870272 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.870243    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhhd\" (UniqueName: \"kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd\") pod \"maas-api-revoke-sub-1297dd90-9grc7\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:47:59.972690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:47:59.972656    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:48:00.281112 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:00.281089    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"]
Nov 02 14:48:00.283653 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:48:00.283622    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2acbec_531d_4dec_aa7c_35ad15ac8336.slice/crio-f93453988167f0e86d0d594872cb536d4e9922f1af1698b14985eca68505052c WatchSource:0}: Error finding container f93453988167f0e86d0d594872cb536d4e9922f1af1698b14985eca68505052c: Status 404 returned error can't find the container with id f93453988167f0e86d0d594872cb536d4e9922f1af1698b14985eca68505052c
Nov 02 14:48:01.035335 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:01.035276    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7" event={"ID":"8d2acbec-531d-4dec-aa7c-35ad15ac8336","Type":"ContainerStarted","Data":"7b17b287ba3d71e8fb8be75aa9fb010249aa364ee71e98ee490fdb40d359de73"}
Nov 02 14:48:01.035902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:01.035408    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7" event={"ID":"8d2acbec-531d-4dec-aa7c-35ad15ac8336","Type":"ContainerStarted","Data":"f93453988167f0e86d0d594872cb536d4e9922f1af1698b14985eca68505052c"}
Nov 02 14:48:01.083076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:01.082995    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7" podStartSLOduration=2.082973051 podStartE2EDuration="2.082973051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:01.072439158 +0000 UTC m=+3045.616154754" watchObservedRunningTime="2026-11-02 14:48:01.082973051 +0000 UTC m=+3045.626688648"
Nov 02 14:48:02.088393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:02.082573    4492 generic.go:356] "Generic (PLEG): container finished" podID="8d2acbec-531d-4dec-aa7c-35ad15ac8336" containerID="7b17b287ba3d71e8fb8be75aa9fb010249aa364ee71e98ee490fdb40d359de73" exitCode=0
Nov 02 14:48:02.088393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:02.082650    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7" event={"ID":"8d2acbec-531d-4dec-aa7c-35ad15ac8336","Type":"ContainerDied","Data":"7b17b287ba3d71e8fb8be75aa9fb010249aa364ee71e98ee490fdb40d359de73"}
Nov 02 14:48:03.931450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:03.928901    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:48:03.998094 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:03.998062    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca\") pod \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") "
Nov 02 14:48:03.998269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:03.998114    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca\") pod \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") "
Nov 02 14:48:03.998269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:03.998206    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd\" (UniqueName: \"kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd\") pod \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\" (UID: \"8d2acbec-531d-4dec-aa7c-35ad15ac8336\") "
Nov 02 14:48:03.998600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:03.998574    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca" pod "8d2acbec-531d-4dec-aa7c-35ad15ac8336" (UID: "8d2acbec-531d-4dec-aa7c-35ad15ac8336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:04.000527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.000495    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca" pod "8d2acbec-531d-4dec-aa7c-35ad15ac8336" (UID: "8d2acbec-531d-4dec-aa7c-35ad15ac8336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:04.000643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.000594    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd" pod "8d2acbec-531d-4dec-aa7c-35ad15ac8336" (UID: "8d2acbec-531d-4dec-aa7c-35ad15ac8336"). InnerVolumeSpecName "kube-api-access-mkhhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:04.099416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.099385    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:04.099416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.099414    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2acbec-531d-4dec-aa7c-35ad15ac8336-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:04.099416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.099425    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhhd\" (UniqueName: \"kubernetes.io/projected/8d2acbec-531d-4dec-aa7c-35ad15ac8336-kube-api-access-mkhhd\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:04.128841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.128763    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"
Nov 02 14:48:04.128841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.128799    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7" event={"ID":"8d2acbec-531d-4dec-aa7c-35ad15ac8336","Type":"ContainerDied","Data":"f93453988167f0e86d0d594872cb536d4e9922f1af1698b14985eca68505052c"}
Nov 02 14:48:04.129027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:04.128853    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93453988167f0e86d0d594872cb536d4e9922f1af1698b14985eca68505052c"
Nov 02 14:48:10.052898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.052858    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"]
Nov 02 14:48:10.053555 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.053536    4492 cpu_manager.go:414] "containerMap: removing container" podUID="8d2acbec-531d-4dec-aa7c-35ad15ac8336" containerName="revoke-keys"
Nov 02 14:48:10.053657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.053559    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2acbec-531d-4dec-aa7c-35ad15ac8336" containerName="revoke-keys"
Nov 02 14:48:10.053716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.053673    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2acbec-531d-4dec-aa7c-35ad15ac8336" containerName="revoke-keys"
Nov 02 14:48:10.057557 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.057535    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.074724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.074693    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"]
Nov 02 14:48:10.080491 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.080462    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:48:10.080709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.080460    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:48:10.080709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.080499    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:48:10.080709 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.080502    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:48:10.161612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.161583    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.161807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.161646    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.161807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.161762    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.263192 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.263148    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.263443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.263209    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.263443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.263288    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.263944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.263921    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.265738 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.265715    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.284712 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.284682    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp\") pod \"maas-api-revoke-sub-7d03a63c-7cxwx\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.373901 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.373804    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:10.675874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:10.675843    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"]
Nov 02 14:48:10.679447 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:48:10.679419    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b0f4f2_5140_480b_8328_984fe13f7bcd.slice/crio-58fcce2a4422a341cefa5978994704b53db5879c671ef5ffb56d1ee225ade556 WatchSource:0}: Error finding container 58fcce2a4422a341cefa5978994704b53db5879c671ef5ffb56d1ee225ade556: Status 404 returned error can't find the container with id 58fcce2a4422a341cefa5978994704b53db5879c671ef5ffb56d1ee225ade556
Nov 02 14:48:11.083804 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:11.083765    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:48:11.171793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:11.171757    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx" event={"ID":"32b0f4f2-5140-480b-8328-984fe13f7bcd","Type":"ContainerStarted","Data":"922e53ef9c76a877ffc966c97f1fd328a0dae12635a1d8aba699a3b04db04790"}
Nov 02 14:48:11.171979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:11.171818    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx" event={"ID":"32b0f4f2-5140-480b-8328-984fe13f7bcd","Type":"ContainerStarted","Data":"58fcce2a4422a341cefa5978994704b53db5879c671ef5ffb56d1ee225ade556"}
Nov 02 14:48:11.212841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:11.212734    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx" podStartSLOduration=1.212716642 podStartE2EDuration="1.212716642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:11.208212723 +0000 UTC m=+3055.751928330" watchObservedRunningTime="2026-11-02 14:48:11.212716642 +0000 UTC m=+3055.756432244"
Nov 02 14:48:12.183201 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:12.183160    4492 generic.go:356] "Generic (PLEG): container finished" podID="32b0f4f2-5140-480b-8328-984fe13f7bcd" containerID="922e53ef9c76a877ffc966c97f1fd328a0dae12635a1d8aba699a3b04db04790" exitCode=0
Nov 02 14:48:12.183816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:12.183228    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx" event={"ID":"32b0f4f2-5140-480b-8328-984fe13f7bcd","Type":"ContainerDied","Data":"922e53ef9c76a877ffc966c97f1fd328a0dae12635a1d8aba699a3b04db04790"}
Nov 02 14:48:14.009240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.008948    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:14.187437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.171026    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp\") pod \"32b0f4f2-5140-480b-8328-984fe13f7bcd\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") "
Nov 02 14:48:14.187814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.187785    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca\") pod \"32b0f4f2-5140-480b-8328-984fe13f7bcd\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") "
Nov 02 14:48:14.188016 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.187986    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca\") pod \"32b0f4f2-5140-480b-8328-984fe13f7bcd\" (UID: \"32b0f4f2-5140-480b-8328-984fe13f7bcd\") "
Nov 02 14:48:14.200613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.200582    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca" pod "32b0f4f2-5140-480b-8328-984fe13f7bcd" (UID: "32b0f4f2-5140-480b-8328-984fe13f7bcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:14.245402 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.220491    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca" pod "32b0f4f2-5140-480b-8328-984fe13f7bcd" (UID: "32b0f4f2-5140-480b-8328-984fe13f7bcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:14.293517 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.293480    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx" event={"ID":"32b0f4f2-5140-480b-8328-984fe13f7bcd","Type":"ContainerDied","Data":"58fcce2a4422a341cefa5978994704b53db5879c671ef5ffb56d1ee225ade556"}
Nov 02 14:48:14.293843 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.293803    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58fcce2a4422a341cefa5978994704b53db5879c671ef5ffb56d1ee225ade556"
Nov 02 14:48:14.294103 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.294087    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"
Nov 02 14:48:14.303686 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.303660    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:14.303883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.303865    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b0f4f2-5140-480b-8328-984fe13f7bcd-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:14.309657 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.309534    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp" pod "32b0f4f2-5140-480b-8328-984fe13f7bcd" (UID: "32b0f4f2-5140-480b-8328-984fe13f7bcd"). InnerVolumeSpecName "kube-api-access-c92xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:14.405556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:14.405515    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/32b0f4f2-5140-480b-8328-984fe13f7bcd-kube-api-access-c92xp\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:16.516484 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:48:16.516447    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:48:17.495499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:17.495469    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:48:20.487013 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.486971    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"]
Nov 02 14:48:20.487462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.487443    4492 cpu_manager.go:414] "containerMap: removing container" podUID="32b0f4f2-5140-480b-8328-984fe13f7bcd" containerName="revoke-keys"
Nov 02 14:48:20.487462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.487460    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b0f4f2-5140-480b-8328-984fe13f7bcd" containerName="revoke-keys"
Nov 02 14:48:20.487569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.487558    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b0f4f2-5140-480b-8328-984fe13f7bcd" containerName="revoke-keys"
Nov 02 14:48:20.490768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.490750    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.503693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.503669    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:48:20.513390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.513332    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:48:20.513705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.513681    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:48:20.519023 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.513822    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:48:20.519154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.518967    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"]
Nov 02 14:48:20.571805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.571764    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.571805 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.571817    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgv2f\" (UniqueName: \"kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.572031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.571877    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.672605 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.672564    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.672812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.672617    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgv2f\" (UniqueName: \"kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.672812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.672640    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.673389 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.673344    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.675295 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.675259    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.694904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.694877    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgv2f\" (UniqueName: \"kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f\") pod \"maas-api-revoke-sub-cd002848-m7544\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:20.808076 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:20.807989    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:21.123590 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:21.123560    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"]
Nov 02 14:48:21.126393 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:48:21.126324    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cd178e_e30e_4204_97f9_0634df437fef.slice/crio-62c660fd935bdfadb321d1867ef58815c040f97be49542df7141054d1a7b72aa WatchSource:0}: Error finding container 62c660fd935bdfadb321d1867ef58815c040f97be49542df7141054d1a7b72aa: Status 404 returned error can't find the container with id 62c660fd935bdfadb321d1867ef58815c040f97be49542df7141054d1a7b72aa
Nov 02 14:48:21.332978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:21.332936    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544" event={"ID":"53cd178e-e30e-4204-97f9-0634df437fef","Type":"ContainerStarted","Data":"62c660fd935bdfadb321d1867ef58815c040f97be49542df7141054d1a7b72aa"}
Nov 02 14:48:22.344349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:22.344237    4492 generic.go:356] "Generic (PLEG): container finished" podID="53cd178e-e30e-4204-97f9-0634df437fef" containerID="411f672d66088a57a202b9855740bd63605a5c0ef1e951e4c725555495e68317" exitCode=0
Nov 02 14:48:22.344349 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:22.344332    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544" event={"ID":"53cd178e-e30e-4204-97f9-0634df437fef","Type":"ContainerDied","Data":"411f672d66088a57a202b9855740bd63605a5c0ef1e951e4c725555495e68317"}
Nov 02 14:48:24.200986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.200959    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:24.421463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.400218    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f\" (UniqueName: \"kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f\") pod \"53cd178e-e30e-4204-97f9-0634df437fef\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") "
Nov 02 14:48:24.421917 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.421884    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca\") pod \"53cd178e-e30e-4204-97f9-0634df437fef\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") "
Nov 02 14:48:24.422098 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.422080    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca\") pod \"53cd178e-e30e-4204-97f9-0634df437fef\" (UID: \"53cd178e-e30e-4204-97f9-0634df437fef\") "
Nov 02 14:48:24.458753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.442159    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f" pod "53cd178e-e30e-4204-97f9-0634df437fef" (UID: "53cd178e-e30e-4204-97f9-0634df437fef"). InnerVolumeSpecName "kube-api-access-wgv2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:24.458753 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.442564    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca" pod "53cd178e-e30e-4204-97f9-0634df437fef" (UID: "53cd178e-e30e-4204-97f9-0634df437fef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:24.461027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.460995    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544" event={"ID":"53cd178e-e30e-4204-97f9-0634df437fef","Type":"ContainerDied","Data":"62c660fd935bdfadb321d1867ef58815c040f97be49542df7141054d1a7b72aa"}
Nov 02 14:48:24.461240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.461223    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62c660fd935bdfadb321d1867ef58815c040f97be49542df7141054d1a7b72aa"
Nov 02 14:48:24.461434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.461419    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"
Nov 02 14:48:24.493916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.493856    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca" pod "53cd178e-e30e-4204-97f9-0634df437fef" (UID: "53cd178e-e30e-4204-97f9-0634df437fef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:24.524216 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.524121    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgv2f\" (UniqueName: \"kubernetes.io/projected/53cd178e-e30e-4204-97f9-0634df437fef-kube-api-access-wgv2f\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:24.524216 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.524168    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:24.524216 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:24.524184    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cd178e-e30e-4204-97f9-0634df437fef-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:30.936049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.936013    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"]
Nov 02 14:48:30.936558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.936539    4492 cpu_manager.go:414] "containerMap: removing container" podUID="53cd178e-e30e-4204-97f9-0634df437fef" containerName="revoke-keys"
Nov 02 14:48:30.936636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.936561    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cd178e-e30e-4204-97f9-0634df437fef" containerName="revoke-keys"
Nov 02 14:48:30.936673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.936648    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cd178e-e30e-4204-97f9-0634df437fef" containerName="revoke-keys"
Nov 02 14:48:30.940142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.940121    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:30.957988 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.957958    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"]
Nov 02 14:48:30.963746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.963717    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:48:30.963907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.963761    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:48:30.963907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.963763    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:48:30.963907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.963874    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:48:30.998132 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.998092    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:30.998328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.998253    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:30.998328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:30.998302    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hq5\" (UniqueName: \"kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.049028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.048990    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:48:31.099467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.099434    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.099652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.099510    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.099652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.099537    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hq5\" (UniqueName: \"kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.100091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.100068    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.102063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.102036    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.121552 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.121520    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hq5\" (UniqueName: \"kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5\") pod \"maas-api-revoke-sub-568cb5a2-kbzhm\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.257485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.257404    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:31.558530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:31.558448    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"]
Nov 02 14:48:31.563042 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:48:31.563011    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8343c69_a8d8_4232_a3ea_4280d0a95a58.slice/crio-d549843c7dad481a502eabffbdc4f68062e00b38c58014e4dbe8fa53b31db201 WatchSource:0}: Error finding container d549843c7dad481a502eabffbdc4f68062e00b38c58014e4dbe8fa53b31db201: Status 404 returned error can't find the container with id d549843c7dad481a502eabffbdc4f68062e00b38c58014e4dbe8fa53b31db201
Nov 02 14:48:32.512626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:32.512582    4492 generic.go:356] "Generic (PLEG): container finished" podID="d8343c69-a8d8-4232-a3ea-4280d0a95a58" containerID="4c744f89023fef1111a82d40f4b031691c386c96341e3a66a9b4ac4f5ede7678" exitCode=0
Nov 02 14:48:32.513150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:32.512656    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm" event={"ID":"d8343c69-a8d8-4232-a3ea-4280d0a95a58","Type":"ContainerDied","Data":"4c744f89023fef1111a82d40f4b031691c386c96341e3a66a9b4ac4f5ede7678"}
Nov 02 14:48:32.513150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:32.512706    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm" event={"ID":"d8343c69-a8d8-4232-a3ea-4280d0a95a58","Type":"ContainerStarted","Data":"d549843c7dad481a502eabffbdc4f68062e00b38c58014e4dbe8fa53b31db201"}
Nov 02 14:48:34.410612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.410584    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:34.486409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.486348    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca\") pod \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") "
Nov 02 14:48:34.487291 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.487267    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca\") pod \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") "
Nov 02 14:48:34.487952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.487830    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca" pod "d8343c69-a8d8-4232-a3ea-4280d0a95a58" (UID: "d8343c69-a8d8-4232-a3ea-4280d0a95a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:34.488116 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.488098    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5\" (UniqueName: \"kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5\") pod \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\" (UID: \"d8343c69-a8d8-4232-a3ea-4280d0a95a58\") "
Nov 02 14:48:34.489136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.489110    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:34.536443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.530520    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5" pod "d8343c69-a8d8-4232-a3ea-4280d0a95a58" (UID: "d8343c69-a8d8-4232-a3ea-4280d0a95a58"). InnerVolumeSpecName "kube-api-access-j9hq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:34.536443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.534926    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca" pod "d8343c69-a8d8-4232-a3ea-4280d0a95a58" (UID: "d8343c69-a8d8-4232-a3ea-4280d0a95a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:34.627244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.627158    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9hq5\" (UniqueName: \"kubernetes.io/projected/d8343c69-a8d8-4232-a3ea-4280d0a95a58-kube-api-access-j9hq5\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:34.627244 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.627201    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8343c69-a8d8-4232-a3ea-4280d0a95a58-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:34.663966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.663922    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm" event={"ID":"d8343c69-a8d8-4232-a3ea-4280d0a95a58","Type":"ContainerDied","Data":"d549843c7dad481a502eabffbdc4f68062e00b38c58014e4dbe8fa53b31db201"}
Nov 02 14:48:34.664281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.664241    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d549843c7dad481a502eabffbdc4f68062e00b38c58014e4dbe8fa53b31db201"
Nov 02 14:48:34.664547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.664471    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"
Nov 02 14:48:34.948382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.947476    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:48:34.948382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.947561    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:48:34.972542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.972504    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:48:34.972846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:34.972819    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:48:35.039506 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:35.039467    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:48:35.039791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:35.039753    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:48:41.280142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.280099    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:48:41.357245 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.357211    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"]
Nov 02 14:48:41.357670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.357655    4492 cpu_manager.go:414] "containerMap: removing container" podUID="d8343c69-a8d8-4232-a3ea-4280d0a95a58" containerName="revoke-keys"
Nov 02 14:48:41.357765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.357671    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8343c69-a8d8-4232-a3ea-4280d0a95a58" containerName="revoke-keys"
Nov 02 14:48:41.357765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.357750    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8343c69-a8d8-4232-a3ea-4280d0a95a58" containerName="revoke-keys"
Nov 02 14:48:41.361403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.361377    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.377790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.377764    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:48:41.387275 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.387239    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:48:41.387466 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.387297    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:48:41.387707 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.387683    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:48:41.391944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.391914    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"]
Nov 02 14:48:41.412852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.412293    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhn7s\" (UniqueName: \"kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.412852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.412397    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.412852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.412551    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.514168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.514120    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.514425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.514193    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhn7s\" (UniqueName: \"kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.514425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.514245    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.514962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.514940    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.516790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.516767    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.535956 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.535879    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhn7s\" (UniqueName: \"kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s\") pod \"maas-api-revoke-sub-59a98859-8g4cv\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.677526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.677493    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:41.904793 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.904700    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:48:41.980534 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:41.980492    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"]
Nov 02 14:48:41.984001 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:48:41.983962    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5581dc4d_a6cc_4db7_b184_92ce1304887a.slice/crio-02d4ee2171d11983dde763c7bcc88f6744851e30167d42c1223b1106a75c1e0a WatchSource:0}: Error finding container 02d4ee2171d11983dde763c7bcc88f6744851e30167d42c1223b1106a75c1e0a: Status 404 returned error can't find the container with id 02d4ee2171d11983dde763c7bcc88f6744851e30167d42c1223b1106a75c1e0a
Nov 02 14:48:42.014583 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:42.014554    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:48:42.717610 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:42.717572    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv" event={"ID":"5581dc4d-a6cc-4db7-b184-92ce1304887a","Type":"ContainerStarted","Data":"c451c629a817765ba468877f0da48b0f8d72774c58a37d2ad1ed5eb7ff4c39d5"}
Nov 02 14:48:42.718197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:42.717639    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv" event={"ID":"5581dc4d-a6cc-4db7-b184-92ce1304887a","Type":"ContainerStarted","Data":"02d4ee2171d11983dde763c7bcc88f6744851e30167d42c1223b1106a75c1e0a"}
Nov 02 14:48:42.761182 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:42.761100    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv" podStartSLOduration=1.7610763600000001 podStartE2EDuration="1.76107636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:42.755303738 +0000 UTC m=+3087.299019333" watchObservedRunningTime="2026-11-02 14:48:42.76107636 +0000 UTC m=+3087.304791955"
Nov 02 14:48:43.726266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:43.726224    4492 generic.go:356] "Generic (PLEG): container finished" podID="5581dc4d-a6cc-4db7-b184-92ce1304887a" containerID="c451c629a817765ba468877f0da48b0f8d72774c58a37d2ad1ed5eb7ff4c39d5" exitCode=0
Nov 02 14:48:43.726731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:43.726311    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv" event={"ID":"5581dc4d-a6cc-4db7-b184-92ce1304887a","Type":"ContainerDied","Data":"c451c629a817765ba468877f0da48b0f8d72774c58a37d2ad1ed5eb7ff4c39d5"}
Nov 02 14:48:45.498755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.498717    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:45.565907 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.565858    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca\") pod \"5581dc4d-a6cc-4db7-b184-92ce1304887a\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") "
Nov 02 14:48:45.567588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.566339    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s\" (UniqueName: \"kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s\") pod \"5581dc4d-a6cc-4db7-b184-92ce1304887a\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") "
Nov 02 14:48:45.620411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.594076    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca\") pod \"5581dc4d-a6cc-4db7-b184-92ce1304887a\" (UID: \"5581dc4d-a6cc-4db7-b184-92ce1304887a\") "
Nov 02 14:48:45.620411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.612616    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca" pod "5581dc4d-a6cc-4db7-b184-92ce1304887a" (UID: "5581dc4d-a6cc-4db7-b184-92ce1304887a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:45.698379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.679805    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca" pod "5581dc4d-a6cc-4db7-b184-92ce1304887a" (UID: "5581dc4d-a6cc-4db7-b184-92ce1304887a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:45.698379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.681903    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s" pod "5581dc4d-a6cc-4db7-b184-92ce1304887a" (UID: "5581dc4d-a6cc-4db7-b184-92ce1304887a"). InnerVolumeSpecName "kube-api-access-qhn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:45.698379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.698273    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.698379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.698302    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5581dc4d-a6cc-4db7-b184-92ce1304887a-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.698379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.698319    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhn7s\" (UniqueName: \"kubernetes.io/projected/5581dc4d-a6cc-4db7-b184-92ce1304887a-kube-api-access-qhn7s\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.761720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.761675    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv" event={"ID":"5581dc4d-a6cc-4db7-b184-92ce1304887a","Type":"ContainerDied","Data":"02d4ee2171d11983dde763c7bcc88f6744851e30167d42c1223b1106a75c1e0a"}
Nov 02 14:48:45.761919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.761749    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d4ee2171d11983dde763c7bcc88f6744851e30167d42c1223b1106a75c1e0a"
Nov 02 14:48:45.761919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:45.761691    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"
Nov 02 14:48:46.516824 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:48:46.516787    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:48:51.725455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.725414    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"]
Nov 02 14:48:51.727888 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.725875    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5581dc4d-a6cc-4db7-b184-92ce1304887a" containerName="revoke-keys"
Nov 02 14:48:51.727888 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.725887    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5581dc4d-a6cc-4db7-b184-92ce1304887a" containerName="revoke-keys"
Nov 02 14:48:51.727888 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.725946    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5581dc4d-a6cc-4db7-b184-92ce1304887a" containerName="revoke-keys"
Nov 02 14:48:51.728997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.728978    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.749009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.747956    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"]
Nov 02 14:48:51.753874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.753840    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:48:51.754087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.753842    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:48:51.754087 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.753897    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:48:51.754305 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.753948    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:48:51.862012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.861974    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsw6\" (UniqueName: \"kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.862012 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.862023    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.862266 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.862178    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.963527 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.963493    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.963711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.963581    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsw6\" (UniqueName: \"kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.963711 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.963603    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.964150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.964124    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.966118 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.966094    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:51.985938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:51.985868    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsw6\" (UniqueName: \"kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6\") pod \"maas-api-revoke-sub-2c5bd416-48hb2\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:52.042932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:52.042891    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:52.350047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:52.349948    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"]
Nov 02 14:48:52.357398 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:48:52.357340    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ed46da_3b37_4619_a5e4_589e1d59462b.slice/crio-1c1d5ac7b1626a37ed550ff42032d024e3b0ce1c1cc86103761af69ff270aacc WatchSource:0}: Error finding container 1c1d5ac7b1626a37ed550ff42032d024e3b0ce1c1cc86103761af69ff270aacc: Status 404 returned error can't find the container with id 1c1d5ac7b1626a37ed550ff42032d024e3b0ce1c1cc86103761af69ff270aacc
Nov 02 14:48:52.807898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:52.807863    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2" event={"ID":"09ed46da-3b37-4619-a5e4-589e1d59462b","Type":"ContainerStarted","Data":"e106dcc55e255cb323722851f7e9cdc2f6b8bfa7a03862abfe19e4af48aaff50"}
Nov 02 14:48:52.808415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:52.807916    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2" event={"ID":"09ed46da-3b37-4619-a5e4-589e1d59462b","Type":"ContainerStarted","Data":"1c1d5ac7b1626a37ed550ff42032d024e3b0ce1c1cc86103761af69ff270aacc"}
Nov 02 14:48:52.847381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:52.847299    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2" podStartSLOduration=1.847280879 podStartE2EDuration="1.847280879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:52.84688423 +0000 UTC m=+3097.390599824" watchObservedRunningTime="2026-11-02 14:48:52.847280879 +0000 UTC m=+3097.390996476"
Nov 02 14:48:53.820258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:53.820201    4492 generic.go:356] "Generic (PLEG): container finished" podID="09ed46da-3b37-4619-a5e4-589e1d59462b" containerID="e106dcc55e255cb323722851f7e9cdc2f6b8bfa7a03862abfe19e4af48aaff50" exitCode=0
Nov 02 14:48:53.820817 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:53.820413    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2" event={"ID":"09ed46da-3b37-4619-a5e4-589e1d59462b","Type":"ContainerDied","Data":"e106dcc55e255cb323722851f7e9cdc2f6b8bfa7a03862abfe19e4af48aaff50"}
Nov 02 14:48:55.622832 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.622802    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:48:55.736383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.732482    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca\") pod \"09ed46da-3b37-4619-a5e4-589e1d59462b\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") "
Nov 02 14:48:55.736383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.732592    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6\" (UniqueName: \"kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6\") pod \"09ed46da-3b37-4619-a5e4-589e1d59462b\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") "
Nov 02 14:48:55.736383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.732763    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca\") pod \"09ed46da-3b37-4619-a5e4-589e1d59462b\" (UID: \"09ed46da-3b37-4619-a5e4-589e1d59462b\") "
Nov 02 14:48:55.799448 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.751902    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca" pod "09ed46da-3b37-4619-a5e4-589e1d59462b" (UID: "09ed46da-3b37-4619-a5e4-589e1d59462b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:55.821931 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.821894    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca" pod "09ed46da-3b37-4619-a5e4-589e1d59462b" (UID: "09ed46da-3b37-4619-a5e4-589e1d59462b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:55.825264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.824526    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6" pod "09ed46da-3b37-4619-a5e4-589e1d59462b" (UID: "09ed46da-3b37-4619-a5e4-589e1d59462b"). InnerVolumeSpecName "kube-api-access-wgsw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:55.834128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.834046    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsw6\" (UniqueName: \"kubernetes.io/projected/09ed46da-3b37-4619-a5e4-589e1d59462b-kube-api-access-wgsw6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:55.834128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.834085    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:55.834128 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.834102    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ed46da-3b37-4619-a5e4-589e1d59462b-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:48:55.927960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.927912    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2" event={"ID":"09ed46da-3b37-4619-a5e4-589e1d59462b","Type":"ContainerDied","Data":"1c1d5ac7b1626a37ed550ff42032d024e3b0ce1c1cc86103761af69ff270aacc"}
Nov 02 14:48:55.928181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.927985    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1d5ac7b1626a37ed550ff42032d024e3b0ce1c1cc86103761af69ff270aacc"
Nov 02 14:48:55.928181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:48:55.928060    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"
Nov 02 14:49:02.105477 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.105438    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"]
Nov 02 14:49:02.106024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.105875    4492 cpu_manager.go:414] "containerMap: removing container" podUID="09ed46da-3b37-4619-a5e4-589e1d59462b" containerName="revoke-keys"
Nov 02 14:49:02.106024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.105887    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ed46da-3b37-4619-a5e4-589e1d59462b" containerName="revoke-keys"
Nov 02 14:49:02.106024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.105958    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ed46da-3b37-4619-a5e4-589e1d59462b" containerName="revoke-keys"
Nov 02 14:49:02.109455 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.109429    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.133598 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.133562    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:49:02.133814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.133730    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:49:02.133919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.133800    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:49:02.133997 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.133876    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:49:02.138391 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.138336    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"]
Nov 02 14:49:02.203602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.203562    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfzk\" (UniqueName: \"kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.203786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.203625    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.203786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.203754    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.305526 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.305479    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfzk\" (UniqueName: \"kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.305715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.305563    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.305715 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.305621    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.306347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.306326    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.308206 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.308183    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.327580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.327546    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfzk\" (UniqueName: \"kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk\") pod \"maas-api-revoke-sub-4e85605e-rf8g5\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.425095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.425065    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:02.728252 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.728224    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"]
Nov 02 14:49:02.730922 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:49:02.730880    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdce59e_0dfb_4f3c_9a4b_8071ffcfed14.slice/crio-8bff9f7ac0103b9c63ce12e38ebb3971630bf5b57f4e88980acc71d8aec7e64c WatchSource:0}: Error finding container 8bff9f7ac0103b9c63ce12e38ebb3971630bf5b57f4e88980acc71d8aec7e64c: Status 404 returned error can't find the container with id 8bff9f7ac0103b9c63ce12e38ebb3971630bf5b57f4e88980acc71d8aec7e64c
Nov 02 14:49:02.969049 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:02.969014    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5" event={"ID":"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14","Type":"ContainerStarted","Data":"8bff9f7ac0103b9c63ce12e38ebb3971630bf5b57f4e88980acc71d8aec7e64c"}
Nov 02 14:49:03.981005 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:03.980963    4492 generic.go:356] "Generic (PLEG): container finished" podID="9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" containerID="4ddcb0bda292b104a8dfd8b0c9e4e4b7ce0b92d2f0821eacd08958d2fe700cb0" exitCode=0
Nov 02 14:49:03.981558 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:03.981046    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5" event={"ID":"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14","Type":"ContainerDied","Data":"4ddcb0bda292b104a8dfd8b0c9e4e4b7ce0b92d2f0821eacd08958d2fe700cb0"}
Nov 02 14:49:05.717382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.716417    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:05.829803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.828757    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk\" (UniqueName: \"kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk\") pod \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") "
Nov 02 14:49:05.829803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.828839    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca\") pod \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") "
Nov 02 14:49:05.829803 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.828978    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca\") pod \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\" (UID: \"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14\") "
Nov 02 14:49:05.830943 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.830915    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca" pod "9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" (UID: "9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:49:05.928435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.862567    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk" pod "9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" (UID: "9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14"). InnerVolumeSpecName "kube-api-access-5nfzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:05.928435 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.862639    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca" pod "9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" (UID: "9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:49:05.939382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.930096    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:05.998507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:05.998461    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:06.003462 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:06.003425    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfzk\" (UniqueName: \"kubernetes.io/projected/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14-kube-api-access-5nfzk\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:06.032048 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:06.032008    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5" event={"ID":"9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14","Type":"ContainerDied","Data":"8bff9f7ac0103b9c63ce12e38ebb3971630bf5b57f4e88980acc71d8aec7e64c"}
Nov 02 14:49:06.032329 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:06.032309    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bff9f7ac0103b9c63ce12e38ebb3971630bf5b57f4e88980acc71d8aec7e64c"
Nov 02 14:49:06.036693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:06.036667    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"
Nov 02 14:49:09.031186 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:09.031140    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"]
Nov 02 14:49:09.041047 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:09.041017    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4khvt"]
Nov 02 14:49:09.579430 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:09.579397    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc101b13-4831-47cc-ae27-f6533d5b360c" path="/var/lib/kubelet/pods/cc101b13-4831-47cc-ae27-f6533d5b360c/volumes"
Nov 02 14:49:11.215684 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:11.215650    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:12.500289 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.500253    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"]
Nov 02 14:49:12.500727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.500703    4492 cpu_manager.go:414] "containerMap: removing container" podUID="9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" containerName="revoke-keys"
Nov 02 14:49:12.500727 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.500714    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" containerName="revoke-keys"
Nov 02 14:49:12.500802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.500775    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" containerName="revoke-keys"
Nov 02 14:49:12.505164 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.505146    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.529783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.529744    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:49:12.529783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.529770    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:49:12.530043 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.529773    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:49:12.530139 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.530107    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\""
Nov 02 14:49:12.533660 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.533636    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"]
Nov 02 14:49:12.692655 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.692615    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.692876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.692676    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdgn\" (UniqueName: \"kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.692876 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.692772    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.793674 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.793581    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.793674 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.793650    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdgn\" (UniqueName: \"kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.793944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.793682    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.794454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.794432    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.796100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.796080    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.815677 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.815644    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdgn\" (UniqueName: \"kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn\") pod \"maas-api-revoke-sub-493c9d04-78nfb\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:12.819513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:12.819489    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:13.124569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:13.124543    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"]
Nov 02 14:49:13.126476 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:49:13.126447    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1726688e_1552_4214_a05b_846e41bc871e.slice/crio-2799a09d285cd69761efcc784055bf95d36da888c797042803813e67ec6e970d WatchSource:0}: Error finding container 2799a09d285cd69761efcc784055bf95d36da888c797042803813e67ec6e970d: Status 404 returned error can't find the container with id 2799a09d285cd69761efcc784055bf95d36da888c797042803813e67ec6e970d
Nov 02 14:49:14.132884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:14.131645    4492 generic.go:356] "Generic (PLEG): container finished" podID="1726688e-1552-4214-a05b-846e41bc871e" containerID="20caf7e5e2d5f1928af2666f9ebbd5c592311a4046e45dcae2d5292c4b92bb99" exitCode=0
Nov 02 14:49:14.132884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:14.131750    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb" event={"ID":"1726688e-1552-4214-a05b-846e41bc871e","Type":"ContainerDied","Data":"20caf7e5e2d5f1928af2666f9ebbd5c592311a4046e45dcae2d5292c4b92bb99"}
Nov 02 14:49:14.132884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:14.131805    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb" event={"ID":"1726688e-1552-4214-a05b-846e41bc871e","Type":"ContainerStarted","Data":"2799a09d285cd69761efcc784055bf95d36da888c797042803813e67ec6e970d"}
Nov 02 14:49:16.016123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.015786    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:16.109133 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.109100    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn\" (UniqueName: \"kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn\") pod \"1726688e-1552-4214-a05b-846e41bc871e\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") "
Nov 02 14:49:16.109823 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.109799    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca\") pod \"1726688e-1552-4214-a05b-846e41bc871e\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") "
Nov 02 14:49:16.110600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.110582    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca\") pod \"1726688e-1552-4214-a05b-846e41bc871e\" (UID: \"1726688e-1552-4214-a05b-846e41bc871e\") "
Nov 02 14:49:16.111175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.111073    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca" pod "1726688e-1552-4214-a05b-846e41bc871e" (UID: "1726688e-1552-4214-a05b-846e41bc871e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:49:16.111826 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.111805    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1726688e-1552-4214-a05b-846e41bc871e-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:16.112081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.112061    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn" pod "1726688e-1552-4214-a05b-846e41bc871e" (UID: "1726688e-1552-4214-a05b-846e41bc871e"). InnerVolumeSpecName "kube-api-access-qfdgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:16.115898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.115867    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca" pod "1726688e-1552-4214-a05b-846e41bc871e" (UID: "1726688e-1552-4214-a05b-846e41bc871e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:49:16.168314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.168273    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb" event={"ID":"1726688e-1552-4214-a05b-846e41bc871e","Type":"ContainerDied","Data":"2799a09d285cd69761efcc784055bf95d36da888c797042803813e67ec6e970d"}
Nov 02 14:49:16.168314 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.168301    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"
Nov 02 14:49:16.168567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.168386    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2799a09d285cd69761efcc784055bf95d36da888c797042803813e67ec6e970d"
Nov 02 14:49:16.213142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.213106    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1726688e-1552-4214-a05b-846e41bc871e-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:16.213142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:16.213139    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdgn\" (UniqueName: \"kubernetes.io/projected/1726688e-1552-4214-a05b-846e41bc871e-kube-api-access-qfdgn\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:16.516558 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:49:16.516518    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:49:29.304643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.304605    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:29.826368 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.826308    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"]
Nov 02 14:49:29.826849 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.826829    4492 cpu_manager.go:414] "containerMap: removing container" podUID="1726688e-1552-4214-a05b-846e41bc871e" containerName="revoke-keys"
Nov 02 14:49:29.826952 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.826851    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1726688e-1552-4214-a05b-846e41bc871e" containerName="revoke-keys"
Nov 02 14:49:29.827014 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.826991    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="1726688e-1552-4214-a05b-846e41bc871e" containerName="revoke-keys"
Nov 02 14:49:29.831927 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.831875    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.845975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.845948    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-87f75f6f-openshift-default-dockercfg-66sxx\""
Nov 02 14:49:29.848536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.848506    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:29.858814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.858782    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"]
Nov 02 14:49:29.904452 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904403    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904517    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n425w\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904579    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904599    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904672    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904723    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904801    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904839    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.904896 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.904857    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:29.976611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:29.976503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:30.008935 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.008901    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n425w\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.009281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.009257    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.010693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.010645    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.010853 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.010836    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.010982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.010968    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.011779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.011735    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.017421 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.011505    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.017718 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.017698    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.023473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.021594    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.023473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.021636    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.023473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.021756    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.023473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.022150    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.023473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.022390    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.024338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.024316    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.026111 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.026088    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.028690 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.028298    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.049379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.048969    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.052575 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.052542    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n425w\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w\") pod \"e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") " pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.154819 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.154733    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:30.478626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.478599    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"]
Nov 02 14:49:30.480808 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:49:30.480781    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6cd53f_0f30_4f79_bbd5_93277cfc7c70.slice/crio-5db2ad2b279754ef2ec98ab7f5afbba49fb387c23d2937e55b0c9c96adc5180f WatchSource:0}: Error finding container 5db2ad2b279754ef2ec98ab7f5afbba49fb387c23d2937e55b0c9c96adc5180f: Status 404 returned error can't find the container with id 5db2ad2b279754ef2ec98ab7f5afbba49fb387c23d2937e55b0c9c96adc5180f
Nov 02 14:49:30.482959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.482925    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:49:30.483056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.482992    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:49:30.483056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:30.483021    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:49:31.163255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:31.163220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:31.287856 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:31.287818    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" event={"ID":"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70","Type":"ContainerStarted","Data":"edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"}
Nov 02 14:49:31.288092 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:31.287872    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" event={"ID":"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70","Type":"ContainerStarted","Data":"5db2ad2b279754ef2ec98ab7f5afbba49fb387c23d2937e55b0c9c96adc5180f"}
Nov 02 14:49:31.340919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:31.340845    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podStartSLOduration=2.3408267990000002 podStartE2EDuration="2.340826799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:31.33869492 +0000 UTC m=+3135.882410514" watchObservedRunningTime="2026-11-02 14:49:31.340826799 +0000 UTC m=+3135.884542395"
Nov 02 14:49:32.155716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:32.155673    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:32.161152 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:32.161126    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:32.294495 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:32.294464    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:32.295423 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:32.295406    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:33.002197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:33.002157    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:33.132889 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:33.132838    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:34.764492 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:34.764457    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:34.887158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:34.887122    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:35.013320 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:35.013281    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:36.372835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:36.372801    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:36.498300 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:36.498263    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:36.700634 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:36.700602    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:38.599768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:38.599734    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:40.545381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:40.541826    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:41.380898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:41.380870    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:42.430854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:42.430824    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:44.351982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:44.351904    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:44.943336 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:44.943292    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:46.206470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:46.206439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:46.518620 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:49:46.518516    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:49:47.754750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:47.754056    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": read tcp 10.134.0.2:35926->10.134.0.164:15021: read: connection reset by peer" start-of-body=
Nov 02 14:49:47.754750 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:47.754134    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": read tcp 10.134.0.2:35926->10.134.0.164:15021: read: connection reset by peer"
Nov 02 14:49:48.003947 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.003864    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:48.437426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.437392    4492 generic.go:356] "Generic (PLEG): container finished" podID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerID="edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2" exitCode=0
Nov 02 14:49:48.437611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.437457    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" event={"ID":"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70","Type":"ContainerDied","Data":"edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"}
Nov 02 14:49:48.437860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.437839    4492 scope.go:122] "RemoveContainer" containerID="edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"
Nov 02 14:49:48.439122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.439094    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:49:48.439181 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.439162    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:49:48.439215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:48.439197    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:49:49.463513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:49.463473    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" event={"ID":"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70","Type":"ContainerStarted","Data":"ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721"}
Nov 02 14:49:50.058073 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:50.058025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:50.156380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:50.155742    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:50.156380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:50.155866    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 14:49:50.156380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:50.155922    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 14:49:51.156773 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.156734    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 14:49:51.157438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.157397    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 14:49:51.764170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.764135    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"]
Nov 02 14:49:51.767978 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.767961    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.792447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.792244    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:49:51.792447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.792304    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:49:51.792447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.792384    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:49:51.792447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.792404    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:49:51.795993 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.795967    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"]
Nov 02 14:49:51.808496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.808465    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.808652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.808510    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.808652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.808572    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t86z\" (UniqueName: \"kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.909340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.909306    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.909340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.909347    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.909605 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.909421    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t86z\" (UniqueName: \"kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.910123 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.910103    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.911846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.911830    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.956381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.955551    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t86z\" (UniqueName: \"kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z\") pod \"maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:51.959395 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:51.959323    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:52.081764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:52.081673    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:52.163535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:52.162801    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 14:49:52.163535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:52.162854    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 14:49:52.410072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:52.410045    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"]
Nov 02 14:49:52.412722 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:49:52.412677    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0796c2_1923_4df2_9ed6_6474d01fb367.slice/crio-9247985cd39ca18b1df8a90162919b101031574aad5959593e1c4717d20bfb3c WatchSource:0}: Error finding container 9247985cd39ca18b1df8a90162919b101031574aad5959593e1c4717d20bfb3c: Status 404 returned error can't find the container with id 9247985cd39ca18b1df8a90162919b101031574aad5959593e1c4717d20bfb3c
Nov 02 14:49:52.487960 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:52.487928    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m" event={"ID":"cd0796c2-1923-4df2-9ed6-6474d01fb367","Type":"ContainerStarted","Data":"9247985cd39ca18b1df8a90162919b101031574aad5959593e1c4717d20bfb3c"}
Nov 02 14:49:53.157877 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:53.157826    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 14:49:53.158101 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:53.157922    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 14:49:53.495482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:53.495451    4492 generic.go:356] "Generic (PLEG): container finished" podID="cd0796c2-1923-4df2-9ed6-6474d01fb367" containerID="addf4172f339115eb6e05e525e1b8f4ea43f6a1a95e844fa2855eba3d091763d" exitCode=0
Nov 02 14:49:53.495873 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:53.495536    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m" event={"ID":"cd0796c2-1923-4df2-9ed6-6474d01fb367","Type":"ContainerDied","Data":"addf4172f339115eb6e05e525e1b8f4ea43f6a1a95e844fa2855eba3d091763d"}
Nov 02 14:49:53.826015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:53.825933    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:54.156437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.155788    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 14:49:54.156437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.155852    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 14:49:54.642322 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.642296    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:54.740485 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.740453    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca\") pod \"cd0796c2-1923-4df2-9ed6-6474d01fb367\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") "
Nov 02 14:49:54.740693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.740519    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z\" (UniqueName: \"kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z\") pod \"cd0796c2-1923-4df2-9ed6-6474d01fb367\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") "
Nov 02 14:49:54.740693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.740584    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca\") pod \"cd0796c2-1923-4df2-9ed6-6474d01fb367\" (UID: \"cd0796c2-1923-4df2-9ed6-6474d01fb367\") "
Nov 02 14:49:54.740967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.740940    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca" pod "cd0796c2-1923-4df2-9ed6-6474d01fb367" (UID: "cd0796c2-1923-4df2-9ed6-6474d01fb367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:49:54.742704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.742681    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca" pod "cd0796c2-1923-4df2-9ed6-6474d01fb367" (UID: "cd0796c2-1923-4df2-9ed6-6474d01fb367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:49:54.742816 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.742802    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z" pod "cd0796c2-1923-4df2-9ed6-6474d01fb367" (UID: "cd0796c2-1923-4df2-9ed6-6474d01fb367"). InnerVolumeSpecName "kube-api-access-5t86z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:54.841614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.841525    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t86z\" (UniqueName: \"kubernetes.io/projected/cd0796c2-1923-4df2-9ed6-6474d01fb367-kube-api-access-5t86z\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:54.841614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.841559    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:54.841614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:54.841569    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd0796c2-1923-4df2-9ed6-6474d01fb367-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:49:55.159029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.158950    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:55.159432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.159410    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:55.160141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.160125    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:49:55.508854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.508826    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"
Nov 02 14:49:55.509052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.508862    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m" event={"ID":"cd0796c2-1923-4df2-9ed6-6474d01fb367","Type":"ContainerDied","Data":"9247985cd39ca18b1df8a90162919b101031574aad5959593e1c4717d20bfb3c"}
Nov 02 14:49:55.509052 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.508910    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9247985cd39ca18b1df8a90162919b101031574aad5959593e1c4717d20bfb3c"
Nov 02 14:49:55.616719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:55.616679    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:57.408959 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:57.408920    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:59.222200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:59.222167    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:59.459696 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:59.459661    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:49:59.613310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:49:59.613201    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:00.797670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:00.797639    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:01.494416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:01.494386    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:01.610915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:01.610859    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:01.648828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:01.648790    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"]
Nov 02 14:50:01.656612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:01.656573    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87f75f6f-6n75m"]
Nov 02 14:50:03.580104 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:03.580071    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0796c2-1923-4df2-9ed6-6474d01fb367" path="/var/lib/kubelet/pods/cd0796c2-1923-4df2-9ed6-6474d01fb367/volumes"
Nov 02 14:50:05.170213 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:05.170169    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"]
Nov 02 14:50:05.170731 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:05.170586    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" containerID="cri-o://ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721" gracePeriod=30
Nov 02 14:50:05.225219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:05.225182    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:06.661474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:06.661439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:10.167471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.167427    4492 patch_prober.go:29] interesting pod/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:50:10.168088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.168052    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:50:10.616682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.616655    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:50:10.639315 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.639280    4492 generic.go:356] "Generic (PLEG): container finished" podID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerID="ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721" exitCode=0
Nov 02 14:50:10.639504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.639326    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" event={"ID":"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70","Type":"ContainerDied","Data":"ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721"}
Nov 02 14:50:10.639504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.639398    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc" event={"ID":"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70","Type":"ContainerDied","Data":"5db2ad2b279754ef2ec98ab7f5afbba49fb387c23d2937e55b0c9c96adc5180f"}
Nov 02 14:50:10.639504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.639409    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"
Nov 02 14:50:10.639504 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.639422    4492 scope.go:122] "RemoveContainer" containerID="ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721"
Nov 02 14:50:10.649416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.649393    4492 scope.go:122] "RemoveContainer" containerID="edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"
Nov 02 14:50:10.658716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.658686    4492 scope.go:122] "RemoveContainer" containerID="ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721"
Nov 02 14:50:10.659092 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:50:10.659067    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721\": container with ID starting with ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721 not found: ID does not exist" containerID="ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721"
Nov 02 14:50:10.659183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.659099    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721"} err="failed to get container status \"ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721\": rpc error: code = NotFound desc = could not find container \"ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721\": container with ID starting with ff6da6107b4e11831dd5fbe00f28f7a50d0cf57313e3aeb9484204dd80b6b721 not found: ID does not exist"
Nov 02 14:50:10.659183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.659119    4492 scope.go:122] "RemoveContainer" containerID="edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"
Nov 02 14:50:10.659410 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:50:10.659390    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2\": container with ID starting with edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2 not found: ID does not exist" containerID="edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"
Nov 02 14:50:10.659513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.659417    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2"} err="failed to get container status \"edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2\": rpc error: code = NotFound desc = could not find container \"edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2\": container with ID starting with edf7c80c90d8264056542eafcbe760cb5514aefec19796fa76aea4e840a9c7f2 not found: ID does not exist"
Nov 02 14:50:10.669992 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.669971    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670032    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670068    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670109    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670141    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670215 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670187    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670220    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670261    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670337    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert\") pod \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\" (UID: \"ca6cd53f-0f30-4f79-bbd5-93277cfc7c70\") "
Nov 02 14:50:10.670569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670424    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:10.670569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670439    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:10.670697 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670673    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:10.670754 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670700    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.670754 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670723    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.670828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670759    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:10.670882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.670847    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:10.672525 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.672490    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:10.672700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.672679    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "kube-api-access-n425w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:10.673146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.673126    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:10.673146 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.673131    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo" pod "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" (UID: "ca6cd53f-0f30-4f79-bbd5-93277cfc7c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772061    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772100    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n425w\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-kube-api-access-n425w\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772113    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772122    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772129    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772137    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.772158 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.772144    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:10.972768 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:50:10.972738    4492 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6cd53f_0f30_4f79_bbd5_93277cfc7c70.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 14:50:10.972899 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:50:10.972850    4492 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6cd53f_0f30_4f79_bbd5_93277cfc7c70.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 14:50:10.992898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:10.992863    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"]
Nov 02 14:50:11.002586 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.002555    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-87f75f6f-openshift-default-5fc6745d5f-jjvcc"]
Nov 02 14:50:11.327481 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.327451    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:11.583011 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.582910    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" path="/var/lib/kubelet/pods/ca6cd53f-0f30-4f79-bbd5-93277cfc7c70/volumes"
Nov 02 14:50:11.839236 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839150    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"]
Nov 02 14:50:11.839733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839712    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy"
Nov 02 14:50:11.839733 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839732    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy"
Nov 02 14:50:11.839925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839742    4492 cpu_manager.go:414] "containerMap: removing container" podUID="cd0796c2-1923-4df2-9ed6-6474d01fb367" containerName="revoke-keys"
Nov 02 14:50:11.839925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839751    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0796c2-1923-4df2-9ed6-6474d01fb367" containerName="revoke-keys"
Nov 02 14:50:11.839925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839771    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy"
Nov 02 14:50:11.839925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839776    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy"
Nov 02 14:50:11.839925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839866    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy"
Nov 02 14:50:11.839925 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.839874    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0796c2-1923-4df2-9ed6-6474d01fb367" containerName="revoke-keys"
Nov 02 14:50:11.844463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.844446    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.858946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.858923    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:50:11.868918 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.868895    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:50:11.869072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.868961    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:50:11.869127 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.868964    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:50:11.872105 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.872082    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"]
Nov 02 14:50:11.882681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.882656    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.882788 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.882713    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgrj\" (UniqueName: \"kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.882841 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.882800    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.983866 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.983832    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgrj\" (UniqueName: \"kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.984100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.983885    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.984100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.984025    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.984659 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.984637    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:11.986544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:11.986523    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:12.006183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:12.006148    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgrj\" (UniqueName: \"kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj\") pod \"maas-api-revoke-sub-f904bff9-szc2s\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:12.157868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:12.157777    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:12.462834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:12.462798    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"]
Nov 02 14:50:12.467082 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:50:12.467056    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde052779_bb3d_4027_9225_4387fd4ee7d7.slice/crio-c10c4ce5045164ad1b50d06be30b65425dc993ec644e6e36fdbafd1c7b971f21 WatchSource:0}: Error finding container c10c4ce5045164ad1b50d06be30b65425dc993ec644e6e36fdbafd1c7b971f21: Status 404 returned error can't find the container with id c10c4ce5045164ad1b50d06be30b65425dc993ec644e6e36fdbafd1c7b971f21
Nov 02 14:50:12.657572 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:12.657535    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s" event={"ID":"de052779-bb3d-4027-9225-4387fd4ee7d7","Type":"ContainerStarted","Data":"c10c4ce5045164ad1b50d06be30b65425dc993ec644e6e36fdbafd1c7b971f21"}
Nov 02 14:50:13.666547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:13.666436    4492 generic.go:356] "Generic (PLEG): container finished" podID="de052779-bb3d-4027-9225-4387fd4ee7d7" containerID="47e2c8512308ba319196e7f400d9f49c329643e3e1bb467a62435b86b1ebeffa" exitCode=0
Nov 02 14:50:13.666547 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:13.666477    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s" event={"ID":"de052779-bb3d-4027-9225-4387fd4ee7d7","Type":"ContainerDied","Data":"47e2c8512308ba319196e7f400d9f49c329643e3e1bb467a62435b86b1ebeffa"}
Nov 02 14:50:14.813499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.813474    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:14.914646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.914612    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca\") pod \"de052779-bb3d-4027-9225-4387fd4ee7d7\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") "
Nov 02 14:50:14.914814 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.914685    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj\" (UniqueName: \"kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj\") pod \"de052779-bb3d-4027-9225-4387fd4ee7d7\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") "
Nov 02 14:50:14.914878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.914858    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca\") pod \"de052779-bb3d-4027-9225-4387fd4ee7d7\" (UID: \"de052779-bb3d-4027-9225-4387fd4ee7d7\") "
Nov 02 14:50:14.915007 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.914972    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca" pod "de052779-bb3d-4027-9225-4387fd4ee7d7" (UID: "de052779-bb3d-4027-9225-4387fd4ee7d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:14.915306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.915286    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:14.916880 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.916862    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca" pod "de052779-bb3d-4027-9225-4387fd4ee7d7" (UID: "de052779-bb3d-4027-9225-4387fd4ee7d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:50:14.916982 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:14.916888    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj" pod "de052779-bb3d-4027-9225-4387fd4ee7d7" (UID: "de052779-bb3d-4027-9225-4387fd4ee7d7"). InnerVolumeSpecName "kube-api-access-5fgrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:15.016088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:15.016048    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgrj\" (UniqueName: \"kubernetes.io/projected/de052779-bb3d-4027-9225-4387fd4ee7d7-kube-api-access-5fgrj\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:15.016088 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:15.016083    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de052779-bb3d-4027-9225-4387fd4ee7d7-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:15.679488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:15.679459    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"
Nov 02 14:50:15.679668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:15.679491    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s" event={"ID":"de052779-bb3d-4027-9225-4387fd4ee7d7","Type":"ContainerDied","Data":"c10c4ce5045164ad1b50d06be30b65425dc993ec644e6e36fdbafd1c7b971f21"}
Nov 02 14:50:15.679668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:15.679545    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c10c4ce5045164ad1b50d06be30b65425dc993ec644e6e36fdbafd1c7b971f21"
Nov 02 14:50:16.516910 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:50:16.516866    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:50:17.427620 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:17.427589    4492 scope.go:122] "RemoveContainer" containerID="5569210bfc462a9a6229cf0b36372c596beeff057e61b85d8b0eb5cb9a028e7c"
Nov 02 14:50:31.293658 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:31.293577    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:34.947410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:34.947125    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:50:34.947410 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:34.947194    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:50:34.982416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:34.982344    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:50:34.982737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:34.982707    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:50:35.041806 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:35.041767    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:50:35.042171 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:35.042122    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:50:36.769406 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.769368    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"]
Nov 02 14:50:36.769940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.769785    4492 cpu_manager.go:414] "containerMap: removing container" podUID="de052779-bb3d-4027-9225-4387fd4ee7d7" containerName="revoke-keys"
Nov 02 14:50:36.769940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.769797    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de052779-bb3d-4027-9225-4387fd4ee7d7" containerName="revoke-keys"
Nov 02 14:50:36.769940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.769888    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6cd53f-0f30-4f79-bbd5-93277cfc7c70" containerName="istio-proxy"
Nov 02 14:50:36.769940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.769917    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="de052779-bb3d-4027-9225-4387fd4ee7d7" containerName="revoke-keys"
Nov 02 14:50:36.774782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.774760    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.793091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.793060    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"]
Nov 02 14:50:36.797408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.797383    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:50:36.797648 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.797619    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:50:36.798757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.798717    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:50:36.798916 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.798902    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:50:36.869006 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.868972    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.869217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.869035    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.869217 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.869164    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgch7\" (UniqueName: \"kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.969908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.969850    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.970081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.969932    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.970081 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.969955    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgch7\" (UniqueName: \"kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.970705 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.970630    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.972621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.972598    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:36.991746 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:36.991718    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgch7\" (UniqueName: \"kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7\") pod \"maas-api-revoke-sub-f03629db-7cbb9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:37.092972 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:37.092885    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:37.401147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:37.401094    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"]
Nov 02 14:50:37.403834 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:50:37.403803    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893fc577_fb19_4bb4_a772_f9238de593a9.slice/crio-7d2987169ff5c92be10e5f3fea68cd5d435e0983e92ecd45d140410e638a326d WatchSource:0}: Error finding container 7d2987169ff5c92be10e5f3fea68cd5d435e0983e92ecd45d140410e638a326d: Status 404 returned error can't find the container with id 7d2987169ff5c92be10e5f3fea68cd5d435e0983e92ecd45d140410e638a326d
Nov 02 14:50:37.918567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:37.918530    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9" event={"ID":"893fc577-fb19-4bb4-a772-f9238de593a9","Type":"ContainerStarted","Data":"6cde04455836917eaf28622185ff01074b920e6d847a5f775156e42123feafce"}
Nov 02 14:50:37.918994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:37.918593    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9" event={"ID":"893fc577-fb19-4bb4-a772-f9238de593a9","Type":"ContainerStarted","Data":"7d2987169ff5c92be10e5f3fea68cd5d435e0983e92ecd45d140410e638a326d"}
Nov 02 14:50:37.954636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:37.954583    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9" podStartSLOduration=1.9545675839999999 podStartE2EDuration="1.954567584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:50:37.953704522 +0000 UTC m=+3202.497420111" watchObservedRunningTime="2026-11-02 14:50:37.954567584 +0000 UTC m=+3202.498283179"
Nov 02 14:50:38.926060 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:38.926018    4492 generic.go:356] "Generic (PLEG): container finished" podID="893fc577-fb19-4bb4-a772-f9238de593a9" containerID="6cde04455836917eaf28622185ff01074b920e6d847a5f775156e42123feafce" exitCode=0
Nov 02 14:50:38.926533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:38.926104    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9" event={"ID":"893fc577-fb19-4bb4-a772-f9238de593a9","Type":"ContainerDied","Data":"6cde04455836917eaf28622185ff01074b920e6d847a5f775156e42123feafce"}
Nov 02 14:50:40.070710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.070689    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:40.205463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.205365    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7\" (UniqueName: \"kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7\") pod \"893fc577-fb19-4bb4-a772-f9238de593a9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") "
Nov 02 14:50:40.205642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.205493    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca\") pod \"893fc577-fb19-4bb4-a772-f9238de593a9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") "
Nov 02 14:50:40.205642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.205532    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca\") pod \"893fc577-fb19-4bb4-a772-f9238de593a9\" (UID: \"893fc577-fb19-4bb4-a772-f9238de593a9\") "
Nov 02 14:50:40.205980 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.205942    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca" pod "893fc577-fb19-4bb4-a772-f9238de593a9" (UID: "893fc577-fb19-4bb4-a772-f9238de593a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:40.207723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.207699    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca" pod "893fc577-fb19-4bb4-a772-f9238de593a9" (UID: "893fc577-fb19-4bb4-a772-f9238de593a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:50:40.207824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.207771    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7" pod "893fc577-fb19-4bb4-a772-f9238de593a9" (UID: "893fc577-fb19-4bb4-a772-f9238de593a9"). InnerVolumeSpecName "kube-api-access-xgch7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:40.306576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.306539    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:40.306576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.306569    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893fc577-fb19-4bb4-a772-f9238de593a9-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:40.306576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.306580    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgch7\" (UniqueName: \"kubernetes.io/projected/893fc577-fb19-4bb4-a772-f9238de593a9-kube-api-access-xgch7\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:40.939412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.939349    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"
Nov 02 14:50:40.939412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.939384    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9" event={"ID":"893fc577-fb19-4bb4-a772-f9238de593a9","Type":"ContainerDied","Data":"7d2987169ff5c92be10e5f3fea68cd5d435e0983e92ecd45d140410e638a326d"}
Nov 02 14:50:40.939631 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:40.939439    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2987169ff5c92be10e5f3fea68cd5d435e0983e92ecd45d140410e638a326d"
Nov 02 14:50:41.501312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:41.501277    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:43.495685 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:43.495649    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:43.607310 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:43.607277    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:50:46.517801 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:50:46.517762    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:50:51.991384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:51.990480    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:50:51.991384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:51.990573    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:50:53.158124 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.158085    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"]
Nov 02 14:50:53.158566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.158551    4492 cpu_manager.go:414] "containerMap: removing container" podUID="893fc577-fb19-4bb4-a772-f9238de593a9" containerName="revoke-keys"
Nov 02 14:50:53.158612 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.158568    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893fc577-fb19-4bb4-a772-f9238de593a9" containerName="revoke-keys"
Nov 02 14:50:53.158669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.158659    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="893fc577-fb19-4bb4-a772-f9238de593a9" containerName="revoke-keys"
Nov 02 14:50:53.162125 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.162099    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.174977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.174942    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:50:53.184968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.184938    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:50:53.185122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.184979    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:50:53.185122 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.185114    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:50:53.189470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.189442    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"]
Nov 02 14:50:53.280576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.280543    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.280757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.280604    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzj5\" (UniqueName: \"kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.280757 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.280713    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.382436 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.382396    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.382580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.382474    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzj5\" (UniqueName: \"kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.382580 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.382501    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.383209 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.383181    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.384911 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.384890    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.404594 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.404565    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzj5\" (UniqueName: \"kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5\") pod \"maas-api-revoke-sub-0165d7d2-zhlvm\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.476576 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.476545    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:53.778237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:53.778210    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"]
Nov 02 14:50:53.780699 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:50:53.780673    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8f7a06_2fc6_445a_bb20_032a6ee2aa30.slice/crio-3af24ce76434830862fefebcfb9bbfc8073a0ecc9dc45cc15e2ca5df29697d07 WatchSource:0}: Error finding container 3af24ce76434830862fefebcfb9bbfc8073a0ecc9dc45cc15e2ca5df29697d07: Status 404 returned error can't find the container with id 3af24ce76434830862fefebcfb9bbfc8073a0ecc9dc45cc15e2ca5df29697d07
Nov 02 14:50:54.039730 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:54.039700    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm" event={"ID":"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30","Type":"ContainerStarted","Data":"3af24ce76434830862fefebcfb9bbfc8073a0ecc9dc45cc15e2ca5df29697d07"}
Nov 02 14:50:55.046995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:55.046947    4492 generic.go:356] "Generic (PLEG): container finished" podID="5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" containerID="b4ef6c79ea41645da29fc203fb1406a0c5e123cc05e48f414a40a3c71b79dc88" exitCode=0
Nov 02 14:50:55.047445 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:55.047032    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm" event={"ID":"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30","Type":"ContainerDied","Data":"b4ef6c79ea41645da29fc203fb1406a0c5e123cc05e48f414a40a3c71b79dc88"}
Nov 02 14:50:56.191914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.191885    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:56.313670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.313580    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca\") pod \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") "
Nov 02 14:50:56.313670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.313623    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca\") pod \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") "
Nov 02 14:50:56.313871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.313685    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5\" (UniqueName: \"kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5\") pod \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\" (UID: \"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30\") "
Nov 02 14:50:56.314120 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.314089    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca" pod "5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" (UID: "5b8f7a06-2fc6-445a-bb20-032a6ee2aa30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:56.315945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.315920    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca" pod "5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" (UID: "5b8f7a06-2fc6-445a-bb20-032a6ee2aa30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:50:56.316038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.315957    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5" pod "5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" (UID: "5b8f7a06-2fc6-445a-bb20-032a6ee2aa30"). InnerVolumeSpecName "kube-api-access-jkzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:56.415626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.415587    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:56.415626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.415619    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:56.415626 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:56.415632    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzj5\" (UniqueName: \"kubernetes.io/projected/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30-kube-api-access-jkzj5\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:50:57.060637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:57.060609    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"
Nov 02 14:50:57.060821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:57.060642    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm" event={"ID":"5b8f7a06-2fc6-445a-bb20-032a6ee2aa30","Type":"ContainerDied","Data":"3af24ce76434830862fefebcfb9bbfc8073a0ecc9dc45cc15e2ca5df29697d07"}
Nov 02 14:50:57.060821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:50:57.060684    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af24ce76434830862fefebcfb9bbfc8073a0ecc9dc45cc15e2ca5df29697d07"
Nov 02 14:51:04.162682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.162642    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"]
Nov 02 14:51:04.163306 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.163283    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" containerName="revoke-keys"
Nov 02 14:51:04.163405 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.163309    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" containerName="revoke-keys"
Nov 02 14:51:04.163488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.163473    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" containerName="revoke-keys"
Nov 02 14:51:04.166941 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.166921    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.181130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.181091    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:51:04.189752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.189695    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:51:04.189752 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.189696    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:51:04.190034 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.189773    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:51:04.194150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.194123    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"]
Nov 02 14:51:04.194977 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.194953    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn67h\" (UniqueName: \"kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.195096 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.195071    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.195157 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.195109    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.296589 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.296555    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn67h\" (UniqueName: \"kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.296807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.296643    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.296807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.296675    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.297465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.297391    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.299235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.299207    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.318682 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.318653    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn67h\" (UniqueName: \"kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h\") pod \"maas-api-revoke-sub-6b19338f-gxzkj\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.481602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.481570    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:04.785719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:04.785686    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"]
Nov 02 14:51:04.787400 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:51:04.787346    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8322fe1d_7c5a_494b_b53e_f0c5b4a133d3.slice/crio-2bd3c59bd5f45748fc4f8e2c398d6722ab82d1897a46c7d2e2ad21064c3ff48b WatchSource:0}: Error finding container 2bd3c59bd5f45748fc4f8e2c398d6722ab82d1897a46c7d2e2ad21064c3ff48b: Status 404 returned error can't find the container with id 2bd3c59bd5f45748fc4f8e2c398d6722ab82d1897a46c7d2e2ad21064c3ff48b
Nov 02 14:51:05.112142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:05.112100    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj" event={"ID":"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3","Type":"ContainerStarted","Data":"7074700e19fd75edd16acbbf26a0b0f2b32c45e6d86f35eaf2a983fd52e46c54"}
Nov 02 14:51:05.112283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:05.112155    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj" event={"ID":"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3","Type":"ContainerStarted","Data":"2bd3c59bd5f45748fc4f8e2c398d6722ab82d1897a46c7d2e2ad21064c3ff48b"}
Nov 02 14:51:05.149418 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:05.149320    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj" podStartSLOduration=1.149305001 podStartE2EDuration="1.149305001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:51:05.148433079 +0000 UTC m=+3229.692148674" watchObservedRunningTime="2026-11-02 14:51:05.149305001 +0000 UTC m=+3229.693020592"
Nov 02 14:51:06.120716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:06.120681    4492 generic.go:356] "Generic (PLEG): container finished" podID="8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" containerID="7074700e19fd75edd16acbbf26a0b0f2b32c45e6d86f35eaf2a983fd52e46c54" exitCode=0
Nov 02 14:51:06.121138 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:06.120763    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj" event={"ID":"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3","Type":"ContainerDied","Data":"7074700e19fd75edd16acbbf26a0b0f2b32c45e6d86f35eaf2a983fd52e46c54"}
Nov 02 14:51:07.272264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.272235    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:07.334238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.334195    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h\" (UniqueName: \"kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h\") pod \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") "
Nov 02 14:51:07.334470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.334272    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca\") pod \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") "
Nov 02 14:51:07.334470 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.334449    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca\") pod \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\" (UID: \"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3\") "
Nov 02 14:51:07.334694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.334667    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca" pod "8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" (UID: "8322fe1d-7c5a-494b-b53e-f0c5b4a133d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:51:07.334852 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.334833    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:51:07.336630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.336605    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca" pod "8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" (UID: "8322fe1d-7c5a-494b-b53e-f0c5b4a133d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:07.336755 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.336672    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h" pod "8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" (UID: "8322fe1d-7c5a-494b-b53e-f0c5b4a133d3"). InnerVolumeSpecName "kube-api-access-hn67h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:07.435518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.435483    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn67h\" (UniqueName: \"kubernetes.io/projected/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-kube-api-access-hn67h\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:51:07.435518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:07.435518    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:51:08.143231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:08.143201    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"
Nov 02 14:51:08.143453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:08.143247    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj" event={"ID":"8322fe1d-7c5a-494b-b53e-f0c5b4a133d3","Type":"ContainerDied","Data":"2bd3c59bd5f45748fc4f8e2c398d6722ab82d1897a46c7d2e2ad21064c3ff48b"}
Nov 02 14:51:08.143453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:08.143309    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd3c59bd5f45748fc4f8e2c398d6722ab82d1897a46c7d2e2ad21064c3ff48b"
Nov 02 14:51:16.623598 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:51:16.623554    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:51:17.255255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.255213    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:51:17.725399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.698795    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:51:17.820970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.820929    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"]
Nov 02 14:51:17.822059 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.822024    4492 cpu_manager.go:414] "containerMap: removing container" podUID="8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" containerName="revoke-keys"
Nov 02 14:51:17.822247 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.822229    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" containerName="revoke-keys"
Nov 02 14:51:17.822584 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.822568    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" containerName="revoke-keys"
Nov 02 14:51:17.991656 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.991574    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"]
Nov 02 14:51:17.991924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:17.991741    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.029283 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.029252    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-170c8925-kserve-self-signed-certs\""
Nov 02 14:51:18.085994 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.085913    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs\") pod \"e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") " pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.086207 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.086037    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6mz\" (UniqueName: \"kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz\") pod \"e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") " pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.119539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.119497    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-170c8925-epp-sa-dockercfg-bqj7g\""
Nov 02 14:51:18.187548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.187501    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs\") pod \"e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") " pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.187759 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.187605    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6mz\" (UniqueName: \"kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz\") pod \"e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") " pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.190409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.190378    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs\") pod \"e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") " pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.212578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.212539    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6mz\" (UniqueName: \"kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz\") pod \"e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") " pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.361786 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.361697    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:18.663420 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.663386    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"]
Nov 02 14:51:18.665628 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:51:18.665599    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aadf52a_0983_48fc_b1ce_2253b11ec926.slice/crio-f10c98a416c38654c07b68d6d0ded8fc2f7a119307a97ad91e1248ceb98c283e WatchSource:0}: Error finding container f10c98a416c38654c07b68d6d0ded8fc2f7a119307a97ad91e1248ceb98c283e: Status 404 returned error can't find the container with id f10c98a416c38654c07b68d6d0ded8fc2f7a119307a97ad91e1248ceb98c283e
Nov 02 14:51:18.667736 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:18.667716    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:51:19.310883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:19.310838    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" event={"ID":"8aadf52a-0983-48fc-b1ce-2253b11ec926","Type":"ContainerStarted","Data":"f10c98a416c38654c07b68d6d0ded8fc2f7a119307a97ad91e1248ceb98c283e"}
Nov 02 14:51:19.716202 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:19.716162    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:51:21.363836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:21.362726    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" event={"ID":"8aadf52a-0983-48fc-b1ce-2253b11ec926","Type":"ContainerStarted","Data":"f6e49629a32d2cf08079ebc7f1c0babfc50e26df8ffbd1828f278a170a990ab8"}
Nov 02 14:51:21.363836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:21.363771    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:51:31.416397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:31.416325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:51:35.046800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:35.046736    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podStartSLOduration=16.070248807 podStartE2EDuration="18.046720807s" totalImagesPullingTime="1.976472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:51:21.411075302 +0000 UTC m=+3245.954790895" watchObservedRunningTime="2026-11-02 14:51:35.046720807 +0000 UTC m=+3259.590436400"
Nov 02 14:51:35.048482 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:35.048449    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"]
Nov 02 14:51:35.064384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:35.064321    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tvvsz"]
Nov 02 14:51:35.581910 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:35.581853    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b5dab7-0378-47c8-83cf-8328136f7b4e" path="/var/lib/kubelet/pods/00b5dab7-0378-47c8-83cf-8328136f7b4e/volumes"
Nov 02 14:51:41.621058 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:41.621025    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:51:46.516668 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:51:46.516629    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:51:48.345376 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:48.345324    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:51:53.394644 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:51:53.394608    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:52:11.031237 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:11.031201    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"]
Nov 02 14:52:11.042637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:11.042605    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5af728b-65n8l"]
Nov 02 14:52:11.579662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:11.579623    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc414e5-ae0b-4b09-8613-d0bba4791181" path="/var/lib/kubelet/pods/2fc414e5-ae0b-4b09-8613-d0bba4791181/volumes"
Nov 02 14:52:16.471243 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:16.471208    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:16.517268 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:52:16.517229    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:52:20.074441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:20.074403    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"]
Nov 02 14:52:20.079891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:20.079862    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:20.195607 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:20.195531    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:23.397874 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:23.397818    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:24.200379 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:24.196825    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:24.317986 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:24.317914    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:32.032875 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:32.032843    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"]
Nov 02 14:52:32.042669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:32.042631    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac1e13d-5j9pt"]
Nov 02 14:52:33.395079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:33.395044    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:33.580033 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:33.580000    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb778013-1129-4e7d-a691-33fe01b7ad16" path="/var/lib/kubelet/pods/eb778013-1129-4e7d-a691-33fe01b7ad16/volumes"
Nov 02 14:52:35.117782 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:35.117748    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main" containerID="cri-o://f6e49629a32d2cf08079ebc7f1c0babfc50e26df8ffbd1828f278a170a990ab8" gracePeriod=45
Nov 02 14:52:40.537771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:40.537734    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:41.764897 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:41.764829    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:52:42.037290 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:42.037179    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"]
Nov 02 14:52:42.045031 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:42.044978    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b123093-n5gd2"]
Nov 02 14:52:43.395003 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:43.394963    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:43.395726 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:43.395706    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:52:43.584348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:43.584306    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3" path="/var/lib/kubelet/pods/e650c7f2-3d46-4aad-b1ba-bdb51b4a60b3/volumes"
Nov 02 14:52:46.519269 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:52:46.519228    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:52:53.033585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:53.033537    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"]
Nov 02 14:52:53.042846 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:53.042808    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd3b9634-ldx5r"]
Nov 02 14:52:53.394775 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:53.394697    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:53.581494 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:53.581459    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22ddf63-6490-43c0-ac6e-519ab0adcf69" path="/var/lib/kubelet/pods/a22ddf63-6490-43c0-ac6e-519ab0adcf69/volumes"
Nov 02 14:52:57.322720 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:52:57.322689    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:03.403490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:03.403447    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:53:04.042653 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:04.042617    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"]
Nov 02 14:53:04.052191 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:04.052156    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1297dd90-9grc7"]
Nov 02 14:53:05.297263 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.297222    4492 generic.go:356] "Generic (PLEG): container finished" podID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerID="f6e49629a32d2cf08079ebc7f1c0babfc50e26df8ffbd1828f278a170a990ab8" exitCode=0
Nov 02 14:53:05.297800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.297300    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" event={"ID":"8aadf52a-0983-48fc-b1ce-2253b11ec926","Type":"ContainerDied","Data":"f6e49629a32d2cf08079ebc7f1c0babfc50e26df8ffbd1828f278a170a990ab8"}
Nov 02 14:53:05.392574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.392495    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:53:05.533252 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.533212    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs\") pod \"8aadf52a-0983-48fc-b1ce-2253b11ec926\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") "
Nov 02 14:53:05.533497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.533447    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz\" (UniqueName: \"kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz\") pod \"8aadf52a-0983-48fc-b1ce-2253b11ec926\" (UID: \"8aadf52a-0983-48fc-b1ce-2253b11ec926\") "
Nov 02 14:53:05.535765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.535729    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs" pod "8aadf52a-0983-48fc-b1ce-2253b11ec926" (UID: "8aadf52a-0983-48fc-b1ce-2253b11ec926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:05.535883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.535770    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz" pod "8aadf52a-0983-48fc-b1ce-2253b11ec926" (UID: "8aadf52a-0983-48fc-b1ce-2253b11ec926"). InnerVolumeSpecName "kube-api-access-gj6mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:05.581365 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.581312    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2acbec-531d-4dec-aa7c-35ad15ac8336" path="/var/lib/kubelet/pods/8d2acbec-531d-4dec-aa7c-35ad15ac8336/volumes"
Nov 02 14:53:05.635258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.635230    4492 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadf52a-0983-48fc-b1ce-2253b11ec926-tls-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:05.635258 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:05.635258    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6mz\" (UniqueName: \"kubernetes.io/projected/8aadf52a-0983-48fc-b1ce-2253b11ec926-kube-api-access-gj6mz\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:06.304169 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:06.304139    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"
Nov 02 14:53:06.304619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:06.304164    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q" event={"ID":"8aadf52a-0983-48fc-b1ce-2253b11ec926","Type":"ContainerDied","Data":"f10c98a416c38654c07b68d6d0ded8fc2f7a119307a97ad91e1248ceb98c283e"}
Nov 02 14:53:06.304619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:06.304227    4492 scope.go:122] "RemoveContainer" containerID="f6e49629a32d2cf08079ebc7f1c0babfc50e26df8ffbd1828f278a170a990ab8"
Nov 02 14:53:06.351281 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:06.351250    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"]
Nov 02 14:53:06.360838 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:06.360811    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-170c8925-kserve-router-scheduler-569997cfbc-vlx2q"]
Nov 02 14:53:07.579280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:07.579244    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" path="/var/lib/kubelet/pods/8aadf52a-0983-48fc-b1ce-2253b11ec926/volumes"
Nov 02 14:53:14.030338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:14.030301    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"]
Nov 02 14:53:14.039859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:14.039828    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d03a63c-7cxwx"]
Nov 02 14:53:15.579844 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:15.579814    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b0f4f2-5140-480b-8328-984fe13f7bcd" path="/var/lib/kubelet/pods/32b0f4f2-5140-480b-8328-984fe13f7bcd/volumes"
Nov 02 14:53:16.517205 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:53:16.517161    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:53:16.579668 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:16.579631    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:17.519597 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:17.519565    4492 scope.go:122] "RemoveContainer" containerID="5ec3f59559db8b417e966f144e795f22678700b31bac420a60b5899c33acf190"
Nov 02 14:53:17.528975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:17.528953    4492 scope.go:122] "RemoveContainer" containerID="71e643a6ac50d84544db27fb0f86413fa2b49a630e3d300357bc3f1482608d78"
Nov 02 14:53:25.030710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:25.030675    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"]
Nov 02 14:53:25.040077 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:25.040046    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd002848-m7544"]
Nov 02 14:53:25.581344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:25.581307    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cd178e-e30e-4204-97f9-0634df437fef" path="/var/lib/kubelet/pods/53cd178e-e30e-4204-97f9-0634df437fef/volumes"
Nov 02 14:53:26.251904 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.251638    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"]
Nov 02 14:53:26.253142 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.253115    4492 cpu_manager.go:414] "containerMap: removing container" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main"
Nov 02 14:53:26.253309 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.253293    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main"
Nov 02 14:53:26.253563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.253546    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aadf52a-0983-48fc-b1ce-2253b11ec926" containerName="main"
Nov 02 14:53:26.263009 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.262981    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.292857 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.292819    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"]
Nov 02 14:53:26.296242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.295931    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:26.296242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.296004    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:26.296242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.296115    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:53:26.296242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.296198    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:26.353879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.353538    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvcvt\" (UniqueName: \"kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.353879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.353623    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.353879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.353667    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.455235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.455185    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvcvt\" (UniqueName: \"kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.455474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.455264    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.455474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.455310    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.456723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.456246    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.459154 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.459102    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.479069 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.479000    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvcvt\" (UniqueName: \"kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt\") pod \"maas-api-revoke-sub-452b6d7b-bsl92\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.626136 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.626039    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:26.962223 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:26.962184    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"]
Nov 02 14:53:26.977408 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:53:26.967818    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ff27ac_41a6_4fc9_b74a_45e717e744d1.slice/crio-d82e94a22e8c190b5052c98020d347b215d6205b8d4032f29c77eae99698e6fd WatchSource:0}: Error finding container d82e94a22e8c190b5052c98020d347b215d6205b8d4032f29c77eae99698e6fd: Status 404 returned error can't find the container with id d82e94a22e8c190b5052c98020d347b215d6205b8d4032f29c77eae99698e6fd
Nov 02 14:53:27.498382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:27.493329    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92" event={"ID":"f7ff27ac-41a6-4fc9-b74a-45e717e744d1","Type":"ContainerStarted","Data":"f10d7b811542bd28918eb2db25383f6381803e425d2c2c8884e8a58945b5ed6b"}
Nov 02 14:53:27.498382 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:27.493425    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92" event={"ID":"f7ff27ac-41a6-4fc9-b74a-45e717e744d1","Type":"ContainerStarted","Data":"d82e94a22e8c190b5052c98020d347b215d6205b8d4032f29c77eae99698e6fd"}
Nov 02 14:53:28.529443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:28.527283    4492 generic.go:356] "Generic (PLEG): container finished" podID="f7ff27ac-41a6-4fc9-b74a-45e717e744d1" containerID="f10d7b811542bd28918eb2db25383f6381803e425d2c2c8884e8a58945b5ed6b" exitCode=0
Nov 02 14:53:28.529443 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:28.527395    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92" event={"ID":"f7ff27ac-41a6-4fc9-b74a-45e717e744d1","Type":"ContainerDied","Data":"f10d7b811542bd28918eb2db25383f6381803e425d2c2c8884e8a58945b5ed6b"}
Nov 02 14:53:29.730673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.730639    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:29.779151 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.779124    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca\") pod \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") "
Nov 02 14:53:29.779338 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.779322    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt\" (UniqueName: \"kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt\") pod \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") "
Nov 02 14:53:29.779431 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.779383    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca\") pod \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\" (UID: \"f7ff27ac-41a6-4fc9-b74a-45e717e744d1\") "
Nov 02 14:53:29.779797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.779750    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca" pod "f7ff27ac-41a6-4fc9-b74a-45e717e744d1" (UID: "f7ff27ac-41a6-4fc9-b74a-45e717e744d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:29.781496 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.781472    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca" pod "f7ff27ac-41a6-4fc9-b74a-45e717e744d1" (UID: "f7ff27ac-41a6-4fc9-b74a-45e717e744d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:29.781975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.781959    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt" pod "f7ff27ac-41a6-4fc9-b74a-45e717e744d1" (UID: "f7ff27ac-41a6-4fc9-b74a-45e717e744d1"). InnerVolumeSpecName "kube-api-access-vvcvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:29.880474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.880394    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvcvt\" (UniqueName: \"kubernetes.io/projected/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-kube-api-access-vvcvt\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:29.880474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.880421    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:29.880474 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:29.880432    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ff27ac-41a6-4fc9-b74a-45e717e744d1-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:30.546183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:30.546154    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"
Nov 02 14:53:30.546396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:30.546159    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92" event={"ID":"f7ff27ac-41a6-4fc9-b74a-45e717e744d1","Type":"ContainerDied","Data":"d82e94a22e8c190b5052c98020d347b215d6205b8d4032f29c77eae99698e6fd"}
Nov 02 14:53:30.546396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:30.546274    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d82e94a22e8c190b5052c98020d347b215d6205b8d4032f29c77eae99698e6fd"
Nov 02 14:53:35.031447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:35.031411    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"]
Nov 02 14:53:35.041662 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:35.041630    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-568cb5a2-kbzhm"]
Nov 02 14:53:35.580377 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:35.580317    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8343c69-a8d8-4232-a3ea-4280d0a95a58" path="/var/lib/kubelet/pods/d8343c69-a8d8-4232-a3ea-4280d0a95a58/volumes"
Nov 02 14:53:37.674100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:37.674064    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:40.715465 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:40.715275    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:41.302299 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:41.302264    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:41.992293 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:41.992256    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:44.567895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.567854    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"]
Nov 02 14:53:44.568514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.568495    4492 cpu_manager.go:414] "containerMap: removing container" podUID="f7ff27ac-41a6-4fc9-b74a-45e717e744d1" containerName="revoke-keys"
Nov 02 14:53:44.568614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.568518    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ff27ac-41a6-4fc9-b74a-45e717e744d1" containerName="revoke-keys"
Nov 02 14:53:44.568679 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.568651    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ff27ac-41a6-4fc9-b74a-45e717e744d1" containerName="revoke-keys"
Nov 02 14:53:44.577985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.577958    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.589108 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.589077    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"]
Nov 02 14:53:44.615370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.611541    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:44.615370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.611674    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:44.615370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.611776    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:53:44.615370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.611867    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:44.718603 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.718551    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.718821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.718697    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcxs\" (UniqueName: \"kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.718821 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.718784    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.820079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.819974    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.820079 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.820047    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcxs\" (UniqueName: \"kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.820348 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.820086    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.821242 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.821214    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.838800 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.838734    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.843141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.843080    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcxs\" (UniqueName: \"kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs\") pod \"maas-api-revoke-sub-3e31a506-bvr5z\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:44.906056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:44.906007    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:45.040970 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:45.040920    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"]
Nov 02 14:53:45.046878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:45.046463    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a98859-8g4cv"]
Nov 02 14:53:45.247835 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:45.247750    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"]
Nov 02 14:53:45.251171 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:53:45.251130    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd21e720_a86b_4f41_a5c3_6661ff55113c.slice/crio-ebd1cda4f8d8793b85b1f46e3b04be1bb849c30bc077e8d119eb5bc0d6f4c7ec WatchSource:0}: Error finding container ebd1cda4f8d8793b85b1f46e3b04be1bb849c30bc077e8d119eb5bc0d6f4c7ec: Status 404 returned error can't find the container with id ebd1cda4f8d8793b85b1f46e3b04be1bb849c30bc077e8d119eb5bc0d6f4c7ec
Nov 02 14:53:45.608710 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:45.608672    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5581dc4d-a6cc-4db7-b184-92ce1304887a" path="/var/lib/kubelet/pods/5581dc4d-a6cc-4db7-b184-92ce1304887a/volumes"
Nov 02 14:53:45.698719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:45.698670    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z" event={"ID":"fd21e720-a86b-4f41-a5c3-6661ff55113c","Type":"ContainerStarted","Data":"ebd1cda4f8d8793b85b1f46e3b04be1bb849c30bc077e8d119eb5bc0d6f4c7ec"}
Nov 02 14:53:46.724491 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:53:46.724321    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:53:46.735940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:46.735144    4492 generic.go:356] "Generic (PLEG): container finished" podID="fd21e720-a86b-4f41-a5c3-6661ff55113c" containerID="a2b11beb1aa3cfaf3c470776dd9a3dfe543bdb53ca5436b093d04620df977db2" exitCode=0
Nov 02 14:53:46.735940 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:46.735390    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z" event={"ID":"fd21e720-a86b-4f41-a5c3-6661ff55113c","Type":"ContainerDied","Data":"a2b11beb1aa3cfaf3c470776dd9a3dfe543bdb53ca5436b093d04620df977db2"}
Nov 02 14:53:47.901533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:47.901499    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:47.999783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:47.999689    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca\") pod \"fd21e720-a86b-4f41-a5c3-6661ff55113c\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") "
Nov 02 14:53:47.999783 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:47.999771    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca\") pod \"fd21e720-a86b-4f41-a5c3-6661ff55113c\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") "
Nov 02 14:53:47.999983 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:47.999809    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs\" (UniqueName: \"kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs\") pod \"fd21e720-a86b-4f41-a5c3-6661ff55113c\" (UID: \"fd21e720-a86b-4f41-a5c3-6661ff55113c\") "
Nov 02 14:53:48.000238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.000204    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca" pod "fd21e720-a86b-4f41-a5c3-6661ff55113c" (UID: "fd21e720-a86b-4f41-a5c3-6661ff55113c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:48.002145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.002116    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca" pod "fd21e720-a86b-4f41-a5c3-6661ff55113c" (UID: "fd21e720-a86b-4f41-a5c3-6661ff55113c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:48.003002 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.002961    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs" pod "fd21e720-a86b-4f41-a5c3-6661ff55113c" (UID: "fd21e720-a86b-4f41-a5c3-6661ff55113c"). InnerVolumeSpecName "kube-api-access-cbcxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:48.100942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.100901    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:48.100942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.100936    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd21e720-a86b-4f41-a5c3-6661ff55113c-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:48.100942 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.100948    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcxs\" (UniqueName: \"kubernetes.io/projected/fd21e720-a86b-4f41-a5c3-6661ff55113c-kube-api-access-cbcxs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:53:48.751637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.751605    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"
Nov 02 14:53:48.751637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.751618    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z" event={"ID":"fd21e720-a86b-4f41-a5c3-6661ff55113c","Type":"ContainerDied","Data":"ebd1cda4f8d8793b85b1f46e3b04be1bb849c30bc077e8d119eb5bc0d6f4c7ec"}
Nov 02 14:53:48.751863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:48.751660    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd1cda4f8d8793b85b1f46e3b04be1bb849c30bc077e8d119eb5bc0d6f4c7ec"
Nov 02 14:53:53.029316 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:53.029270    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:53:55.036150 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:55.036113    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"]
Nov 02 14:53:55.045790 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:55.045758    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5bd416-48hb2"]
Nov 02 14:53:55.579696 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:53:55.579663    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ed46da-3b37-4619-a5e4-589e1d59462b" path="/var/lib/kubelet/pods/09ed46da-3b37-4619-a5e4-589e1d59462b/volumes"
Nov 02 14:54:05.056219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.056181    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"]
Nov 02 14:54:05.056669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.056615    4492 cpu_manager.go:414] "containerMap: removing container" podUID="fd21e720-a86b-4f41-a5c3-6661ff55113c" containerName="revoke-keys"
Nov 02 14:54:05.056669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.056627    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd21e720-a86b-4f41-a5c3-6661ff55113c" containerName="revoke-keys"
Nov 02 14:54:05.056765 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.056700    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd21e720-a86b-4f41-a5c3-6661ff55113c" containerName="revoke-keys"
Nov 02 14:54:05.060979 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.060961    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.078340 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.078298    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"]
Nov 02 14:54:05.084571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.084537    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:54:05.084571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.084537    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:54:05.084571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.084537    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:54:05.085172 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.085147    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 14:54:05.106571 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.106538    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.106893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.106769    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp8z\" (UniqueName: \"kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.106987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.106910    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.208440 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.208382    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.208652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.208493    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp8z\" (UniqueName: \"kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.208652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.208557    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.209178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.209153    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.210932 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.210908    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.230323 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.230293    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp8z\" (UniqueName: \"kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z\") pod \"maas-api-revoke-sub-25b36506-mphb2\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.374734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.374634    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:05.674401 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.674366    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"]
Nov 02 14:54:05.677927 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:54:05.677890    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af26be2_1d78_4b96_9f1f_d4ef315335b0.slice/crio-e8eb61a62f77b4b62f149c91d8c39842b61592e05a63465c7f53b7b1ddbc8496 WatchSource:0}: Error finding container e8eb61a62f77b4b62f149c91d8c39842b61592e05a63465c7f53b7b1ddbc8496: Status 404 returned error can't find the container with id e8eb61a62f77b4b62f149c91d8c39842b61592e05a63465c7f53b7b1ddbc8496
Nov 02 14:54:05.951077 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.950991    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2" event={"ID":"5af26be2-1d78-4b96-9f1f-d4ef315335b0","Type":"ContainerStarted","Data":"c41c177924f669f3efc7dd3e4b927e31fd2ec7aa107d1a82383c562b8fbe3311"}
Nov 02 14:54:05.951077 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.951048    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2" event={"ID":"5af26be2-1d78-4b96-9f1f-d4ef315335b0","Type":"ContainerStarted","Data":"e8eb61a62f77b4b62f149c91d8c39842b61592e05a63465c7f53b7b1ddbc8496"}
Nov 02 14:54:05.988767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:05.988711    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2" podStartSLOduration=0.988695305 podStartE2EDuration="988.695305ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:05.987482814 +0000 UTC m=+3410.531198407" watchObservedRunningTime="2026-11-02 14:54:05.988695305 +0000 UTC m=+3410.532410895"
Nov 02 14:54:06.032370 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:06.032332    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"]
Nov 02 14:54:06.038842 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:06.038806    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e85605e-rf8g5"]
Nov 02 14:54:06.959507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:06.959466    4492 generic.go:356] "Generic (PLEG): container finished" podID="5af26be2-1d78-4b96-9f1f-d4ef315335b0" containerID="c41c177924f669f3efc7dd3e4b927e31fd2ec7aa107d1a82383c562b8fbe3311" exitCode=0
Nov 02 14:54:06.959915 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:06.959552    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2" event={"ID":"5af26be2-1d78-4b96-9f1f-d4ef315335b0","Type":"ContainerDied","Data":"c41c177924f669f3efc7dd3e4b927e31fd2ec7aa107d1a82383c562b8fbe3311"}
Nov 02 14:54:07.579824 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:07.579786    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14" path="/var/lib/kubelet/pods/9fdce59e-0dfb-4f3c-9a4b-8071ffcfed14/volumes"
Nov 02 14:54:08.105404 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.105348    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:08.237165 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.237071    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca\") pod \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") "
Nov 02 14:54:08.237426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.237166    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z\" (UniqueName: \"kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z\") pod \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") "
Nov 02 14:54:08.237426 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.237301    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca\") pod \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\" (UID: \"5af26be2-1d78-4b96-9f1f-d4ef315335b0\") "
Nov 02 14:54:08.237574 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.237546    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca" pod "5af26be2-1d78-4b96-9f1f-d4ef315335b0" (UID: "5af26be2-1d78-4b96-9f1f-d4ef315335b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:08.237729 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.237714    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:54:08.239542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.239518    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca" pod "5af26be2-1d78-4b96-9f1f-d4ef315335b0" (UID: "5af26be2-1d78-4b96-9f1f-d4ef315335b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:08.239542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.239535    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z" pod "5af26be2-1d78-4b96-9f1f-d4ef315335b0" (UID: "5af26be2-1d78-4b96-9f1f-d4ef315335b0"). InnerVolumeSpecName "kube-api-access-tzp8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:08.338432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.338399    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af26be2-1d78-4b96-9f1f-d4ef315335b0-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:54:08.338432 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.338431    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp8z\" (UniqueName: \"kubernetes.io/projected/5af26be2-1d78-4b96-9f1f-d4ef315335b0-kube-api-access-tzp8z\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:54:08.973480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.973445    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2" event={"ID":"5af26be2-1d78-4b96-9f1f-d4ef315335b0","Type":"ContainerDied","Data":"e8eb61a62f77b4b62f149c91d8c39842b61592e05a63465c7f53b7b1ddbc8496"}
Nov 02 14:54:08.973480 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.973481    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"
Nov 02 14:54:08.973693 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:08.973499    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8eb61a62f77b4b62f149c91d8c39842b61592e05a63465c7f53b7b1ddbc8496"
Nov 02 14:54:16.517938 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:54:16.517886    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:54:16.739678 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:16.739643    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:16.850569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:16.850444    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:17.032292 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.032255    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"]
Nov 02 14:54:17.042519 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.042479    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493c9d04-78nfb"]
Nov 02 14:54:17.201454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.201418    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"]
Nov 02 14:54:17.202045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.202023    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5af26be2-1d78-4b96-9f1f-d4ef315335b0" containerName="revoke-keys"
Nov 02 14:54:17.202045 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.202044    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af26be2-1d78-4b96-9f1f-d4ef315335b0" containerName="revoke-keys"
Nov 02 14:54:17.202176 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.202161    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af26be2-1d78-4b96-9f1f-d4ef315335b0" containerName="revoke-keys"
Nov 02 14:54:17.206231 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.206206    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.225919 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.225889    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b21aae-openshift-default-dockercfg-9j2cq\""
Nov 02 14:54:17.256086 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.256054    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"]
Nov 02 14:54:17.300966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.300884    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334181    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334250    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334286    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334318    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334410    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334541    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bqf\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334577    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334617    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.335380 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.334647    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436185 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436143    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bqf\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436195    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436232    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436257    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436311    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436347    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436423    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436460 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436452    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436486    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436724    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.436885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436791    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.437062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436881    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.437062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.436944    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.437388 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.437328    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.438854 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.438833    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.439062 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.439045    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.460822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.460754    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.460822 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.460811    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bqf\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf\") pod \"e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") " pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.484156 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.484123    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:17.520809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.520774    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:17.562890 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.562854    4492 scope.go:122] "RemoveContainer" containerID="922e53ef9c76a877ffc966c97f1fd328a0dae12635a1d8aba699a3b04db04790"
Nov 02 14:54:17.574046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.574020    4492 scope.go:122] "RemoveContainer" containerID="f6a60e9dd8f392b8bb24f1542cfba9af26bb7706c3747186973604a2bf5bdd7e"
Nov 02 14:54:17.582327 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.582298    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1726688e-1552-4214-a05b-846e41bc871e" path="/var/lib/kubelet/pods/1726688e-1552-4214-a05b-846e41bc871e/volumes"
Nov 02 14:54:17.597813 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.596416    4492 scope.go:122] "RemoveContainer" containerID="7b17b287ba3d71e8fb8be75aa9fb010249aa364ee71e98ee490fdb40d359de73"
Nov 02 14:54:17.608704 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.608678    4492 scope.go:122] "RemoveContainer" containerID="77821bc25e54c8b0bacd927d24c3a4ff38e595bd34f97c7e63e3fa6363b5a348"
Nov 02 14:54:17.619246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.619225    4492 scope.go:122] "RemoveContainer" containerID="6464ba3e42c3f7fdedc3a676f482ebcaae99b3793db8b3b4025f2eea832cd576"
Nov 02 14:54:17.844903 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.844802    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"]
Nov 02 14:54:17.847437 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:54:17.847404    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336cc98d_960d_4b59_9a21_398e177f1924.slice/crio-9434b0990ffa36c8361f47b611a20f07c8cde4bb86341d2230df15ea72aef9e7 WatchSource:0}: Error finding container 9434b0990ffa36c8361f47b611a20f07c8cde4bb86341d2230df15ea72aef9e7: Status 404 returned error can't find the container with id 9434b0990ffa36c8361f47b611a20f07c8cde4bb86341d2230df15ea72aef9e7
Nov 02 14:54:17.849637 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.849602    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:54:17.849735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.849693    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:54:17.849735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:17.849724    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:54:18.035114 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:18.035066    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" event={"ID":"336cc98d-960d-4b59-9a21-398e177f1924","Type":"ContainerStarted","Data":"e41fb4258a0da068938e43a29621f9365894d900cdf690f549fce190cbc8898f"}
Nov 02 14:54:18.035319 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:18.035133    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" event={"ID":"336cc98d-960d-4b59-9a21-398e177f1924","Type":"ContainerStarted","Data":"9434b0990ffa36c8361f47b611a20f07c8cde4bb86341d2230df15ea72aef9e7"}
Nov 02 14:54:18.079592 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:18.079534    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podStartSLOduration=1.079517236 podStartE2EDuration="1.079517236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:18.075673134 +0000 UTC m=+3422.619388727" watchObservedRunningTime="2026-11-02 14:54:18.079517236 +0000 UTC m=+3422.623232830"
Nov 02 14:54:18.520971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:18.520930    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:18.526296 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:18.526269    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:19.040929 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:19.040894    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:19.042046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:19.042022    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:20.617228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:20.617195    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:20.735624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:20.735584    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:22.353063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:22.353017    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:22.491130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:22.491094    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:22.610989 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:22.610889    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:24.303865 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:24.303827    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:26.114188 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:26.114149    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:27.930028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:27.929988    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:29.953085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:29.953050    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:31.880254 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:31.880228    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:33.818415 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:33.818378    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:35.082548 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:35.082505    4492 patch_prober.go:29] interesting pod/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:54:35.082968 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:35.082585    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:54:35.604641 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:35.604600    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:35.714513 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:35.714478    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:35.832398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:35.832319    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:37.160744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.160713    4492 generic.go:356] "Generic (PLEG): container finished" podID="336cc98d-960d-4b59-9a21-398e177f1924" containerID="e41fb4258a0da068938e43a29621f9365894d900cdf690f549fce190cbc8898f" exitCode=0
Nov 02 14:54:37.161197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.160770    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" event={"ID":"336cc98d-960d-4b59-9a21-398e177f1924","Type":"ContainerDied","Data":"e41fb4258a0da068938e43a29621f9365894d900cdf690f549fce190cbc8898f"}
Nov 02 14:54:37.161197 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.161126    4492 scope.go:122] "RemoveContainer" containerID="e41fb4258a0da068938e43a29621f9365894d900cdf690f549fce190cbc8898f"
Nov 02 14:54:37.162397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.162348    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:54:37.162509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.162431    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:54:37.162509 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.162461    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 14:54:37.450636 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:37.450605    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:38.168447 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:38.168413    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" event={"ID":"336cc98d-960d-4b59-9a21-398e177f1924","Type":"ContainerStarted","Data":"232ebd3356aaa8783dd6a489891353b5780466d0193a13050dbc11fff1c92c33"}
Nov 02 14:54:38.521257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:38.521218    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:38.521520 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:38.521494    4492 patch_prober.go:29] interesting pod/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 14:54:38.521623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:38.521554    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 14:54:38.764345 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:38.764309    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:38.958619 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:38.958574    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:39.193175 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.193132    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"]
Nov 02 14:54:39.197065 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.197043    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.208095 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.208076    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"]
Nov 02 14:54:39.210453 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.210395    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b21aae\""
Nov 02 14:54:39.210569 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.210513    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b21aae-dockercfg-xvm4t\""
Nov 02 14:54:39.294042 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.294009    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.294224 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.294066    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntch\" (UniqueName: \"kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.315425 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.315387    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"]
Nov 02 14:54:39.319072 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.319049    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.365409 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.365345    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"]
Nov 02 14:54:39.395029 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.394989    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.395238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.395046    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntch\" (UniqueName: \"kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.395238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.395077    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jl4\" (UniqueName: \"kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4\") pod \"payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.395238 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.395148    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.395820 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.395796    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.417467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.417436    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntch\" (UniqueName: \"kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch\") pod \"payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.496734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.496644    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jl4\" (UniqueName: \"kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4\") pod \"payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.496734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.496714    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.497487 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.497462    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.514221 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.514186    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:39.519859 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.519822    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jl4\" (UniqueName: \"kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4\") pod \"payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.525131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.525102    4492 patch_prober.go:29] interesting pod/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 14:54:39.525621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.525163    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 14:54:39.655995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.655619    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:39.859535 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.859500    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"]
Nov 02 14:54:39.863409 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:54:39.863376    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd228fb_67ba_452f_af5e_6aab6a15b2a4.slice/crio-cadef7a37156433562c4da1ef54b3a06eb7e35190c8fdeb07c8ea9365ff23d7d WatchSource:0}: Error finding container cadef7a37156433562c4da1ef54b3a06eb7e35190c8fdeb07c8ea9365ff23d7d: Status 404 returned error can't find the container with id cadef7a37156433562c4da1ef54b3a06eb7e35190c8fdeb07c8ea9365ff23d7d
Nov 02 14:54:39.966895 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:39.966860    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"]
Nov 02 14:54:39.971589 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:54:39.971562    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4b89e1_4ca8_4008_939d_6201d69bac90.slice/crio-0a26ed4f6280ad27fe91aa353f6aae773bc32bedfd7c95d3c91baa5fd5064f4a WatchSource:0}: Error finding container 0a26ed4f6280ad27fe91aa353f6aae773bc32bedfd7c95d3c91baa5fd5064f4a: Status 404 returned error can't find the container with id 0a26ed4f6280ad27fe91aa353f6aae773bc32bedfd7c95d3c91baa5fd5064f4a
Nov 02 14:54:40.027276 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:40.027235    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:40.199429 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:40.199395    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr" event={"ID":"ec4b89e1-4ca8-4008-939d-6201d69bac90","Type":"ContainerStarted","Data":"0a26ed4f6280ad27fe91aa353f6aae773bc32bedfd7c95d3c91baa5fd5064f4a"}
Nov 02 14:54:40.200721 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:40.200695    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97" event={"ID":"4cd228fb-67ba-452f-af5e-6aab6a15b2a4","Type":"ContainerStarted","Data":"cadef7a37156433562c4da1ef54b3a06eb7e35190c8fdeb07c8ea9365ff23d7d"}
Nov 02 14:54:40.522716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:40.522622    4492 patch_prober.go:29] interesting pod/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 14:54:40.522716 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:40.522683    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 14:54:41.521264 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:41.521222    4492 patch_prober.go:29] interesting pod/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 14:54:41.521735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:41.521283    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 14:54:42.058015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:42.057974    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:42.169669 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:42.169633    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:42.521384 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:42.521329    4492 patch_prober.go:29] interesting pod/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 14:54:42.521764 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:42.521415    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 14:54:43.233951 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.233905    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97" event={"ID":"4cd228fb-67ba-452f-af5e-6aab6a15b2a4","Type":"ContainerStarted","Data":"124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c"}
Nov 02 14:54:43.234879 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.234860    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:43.239390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.237856    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr" event={"ID":"ec4b89e1-4ca8-4008-939d-6201d69bac90","Type":"ContainerStarted","Data":"a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7"}
Nov 02 14:54:43.239390 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.237924    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:43.273177 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.273114    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97" podStartSLOduration=1.78481382 podStartE2EDuration="4.273096056s" totalImagesPullingTime="2.488282236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:43.272073494 +0000 UTC m=+3447.815789092" watchObservedRunningTime="2026-11-02 14:54:43.273096056 +0000 UTC m=+3447.816811651"
Nov 02 14:54:43.310672 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.310619    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr" podStartSLOduration=1.706089932 podStartE2EDuration="4.310604527s" totalImagesPullingTime="2.604514595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:43.309526704 +0000 UTC m=+3447.853242300" watchObservedRunningTime="2026-11-02 14:54:43.310604527 +0000 UTC m=+3447.854320120"
Nov 02 14:54:43.525611 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.525526    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:43.526041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.525806    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:43.526588 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:43.526568    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:54:44.395230 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:44.395197    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:46.318501 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:46.318455    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:46.515834 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:54:46.515795    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:54:48.591063 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:48.591031    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:50.420411 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:50.420338    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:52.787417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:52.787381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:54.246174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:54.246146    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:54:54.632900 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:54.632815    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:55.252208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:55.252172    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:54:56.996473 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:56.996393    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:57.463579 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:57.463546    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:57.984061 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:57.984019    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:58.384324 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:58.384244    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:54:58.917799 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:54:58.917766    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:01.330807 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:01.330770    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:02.095882 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:02.095854    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:02.213791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:02.213753    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:03.334827 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:03.334794    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:03.836347 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:03.836312    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:03.950385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:03.950323    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:05.565189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.565152    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"]
Nov 02 14:55:05.569203 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.569185    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.589273 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.589233    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"]
Nov 02 14:55:05.594064 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.594037    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-be6018-serving-cert\""
Nov 02 14:55:05.594235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.594057    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:55:05.594235 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.594114    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:55:05.594398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.594111    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 14:55:05.594398 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.594187    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-245cc\""
Nov 02 14:55:05.647642 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.647612    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.647858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.647775    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqdv\" (UniqueName: \"kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.647858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.647821    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.647963 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.647943    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.653776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.653751    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:05.748834 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.748799    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqdv\" (UniqueName: \"kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.749046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.748847    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.749046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.748908    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.749046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.748938    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.749624 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.749599    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.751737 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.751716    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.751871 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.751852    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.761735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.761676    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:05.770331 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.770311    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqdv\" (UniqueName: \"kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv\") pod \"maas-api-e2e-ipp-b-be6018-69695589c8-d8swn\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:05.883860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:05.883783    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:06.185312 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:06.185245    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"]
Nov 02 14:55:06.188343 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:55:06.188317    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7b73da_1320_4c9e_8483_576d999290d7.slice/crio-9b1f78087328a8274c8a065c25880bec37524f5e1efd6cd5839482cba376c4bb WatchSource:0}: Error finding container 9b1f78087328a8274c8a065c25880bec37524f5e1efd6cd5839482cba376c4bb: Status 404 returned error can't find the container with id 9b1f78087328a8274c8a065c25880bec37524f5e1efd6cd5839482cba376c4bb
Nov 02 14:55:06.380533 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:06.380493    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn" event={"ID":"4b7b73da-1320-4c9e-8483-576d999290d7","Type":"ContainerStarted","Data":"9b1f78087328a8274c8a065c25880bec37524f5e1efd6cd5839482cba376c4bb"}
Nov 02 14:55:07.388921 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:07.388880    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn" event={"ID":"4b7b73da-1320-4c9e-8483-576d999290d7","Type":"ContainerStarted","Data":"61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec"}
Nov 02 14:55:07.389413 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:07.388951    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:07.427166 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:07.427114    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn" podStartSLOduration=2.026500923 podStartE2EDuration="2.427100641s" totalImagesPullingTime="400.599718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:07.4266484 +0000 UTC m=+3471.970364010" watchObservedRunningTime="2026-11-02 14:55:07.427100641 +0000 UTC m=+3471.970816269"
Nov 02 14:55:07.592833 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:07.592803    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:09.429647 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:09.429616    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:11.790015 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:11.789975    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:13.400836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:13.400806    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:55:13.601438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:13.601406    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:15.030944 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:15.030903    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"]
Nov 02 14:55:15.040328 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:15.040293    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-szc2s"]
Nov 02 14:55:15.433368 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:15.433331    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:15.580646 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:15.580612    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de052779-bb3d-4027-9225-4387fd4ee7d7" path="/var/lib/kubelet/pods/de052779-bb3d-4027-9225-4387fd4ee7d7/volumes"
Nov 02 14:55:16.517920 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:55:16.517863    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:55:16.828566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:16.828482    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:17.662524 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.662491    4492 scope.go:122] "RemoveContainer" containerID="20caf7e5e2d5f1928af2666f9ebbd5c592311a4046e45dcae2d5292c4b92bb99"
Nov 02 14:55:17.672068 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.672043    4492 scope.go:122] "RemoveContainer" containerID="4ddcb0bda292b104a8dfd8b0c9e4e4b7ce0b92d2f0821eacd08958d2fe700cb0"
Nov 02 14:55:17.680938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.680914    4492 scope.go:122] "RemoveContainer" containerID="4c744f89023fef1111a82d40f4b031691c386c96341e3a66a9b4ac4f5ede7678"
Nov 02 14:55:17.689801 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.689780    4492 scope.go:122] "RemoveContainer" containerID="e106dcc55e255cb323722851f7e9cdc2f6b8bfa7a03862abfe19e4af48aaff50"
Nov 02 14:55:17.698664 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.698644    4492 scope.go:122] "RemoveContainer" containerID="411f672d66088a57a202b9855740bd63605a5c0ef1e951e4c725555495e68317"
Nov 02 14:55:17.707240 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.707218    4492 scope.go:122] "RemoveContainer" containerID="c451c629a817765ba468877f0da48b0f8d72774c58a37d2ad1ed5eb7ff4c39d5"
Nov 02 14:55:17.797971 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:17.797941    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:19.503394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:19.503315    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:21.537041 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:21.537002    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:22.386600 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:22.386565    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:22.511893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:22.511856    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:22.623051 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:22.623015    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:23.950556 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:23.950521    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:25.490257 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:25.490220    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:25.654393 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:25.654176    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:26.720643 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:26.720608    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:28.541723 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:28.541679    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:30.832585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:30.832553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:33.106193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:33.106156    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:34.947797 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:34.947761    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:37.228499 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:37.228463    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:39.498681 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:39.498649    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:40.033794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:40.033751    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"]
Nov 02 14:55:40.043518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:40.043482    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7cbb9"]
Nov 02 14:55:41.436144 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:41.436108    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:41.580160 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:41.580125    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893fc577-fb19-4bb4-a772-f9238de593a9" path="/var/lib/kubelet/pods/893fc577-fb19-4bb4-a772-f9238de593a9/volumes"
Nov 02 14:55:42.231858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:42.231824    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.811891 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:43.811855    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:46.105456 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:46.105424    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:46.516975 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:55:46.516943    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:55:47.936366 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:47.936312    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:50.330265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:50.330228    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:52.601828 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:52.601800    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:52.730200 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:52.730167    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:54.438265 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:54.438232    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:56.838458 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:56.838424    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:57.032651 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:57.032612    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"]
Nov 02 14:55:57.043794 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:57.043757    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zhlvm"]
Nov 02 14:55:57.580541 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:57.580504    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8f7a06-2fc6-445a-bb20-032a6ee2aa30" path="/var/lib/kubelet/pods/5b8f7a06-2fc6-445a-bb20-032a6ee2aa30/volumes"
Nov 02 14:55:59.210018 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:55:59.209982    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:01.119995 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:01.119936    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:03.539975 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:03.539943    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:05.880735 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:05.880694    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:07.692514 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:07.692478    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:08.030808 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:08.030724    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"]
Nov 02 14:56:08.040161 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:08.040122    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gxzkj"]
Nov 02 14:56:09.581381 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:09.581332    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8322fe1d-7c5a-494b-b53e-f0c5b4a133d3" path="/var/lib/kubelet/pods/8322fe1d-7c5a-494b-b53e-f0c5b4a133d3/volumes"
Nov 02 14:56:10.029623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:10.029561    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:12.390962 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:12.390921    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:14.223147 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:14.223110    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:16.517602 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:56:16.517566    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:56:16.526985 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:16.526950    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:16.956938 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:16.956910    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:17.751885 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:17.751850    4492 scope.go:122] "RemoveContainer" containerID="47e2c8512308ba319196e7f400d9f49c329643e3e1bb467a62435b86b1ebeffa"
Nov 02 14:56:17.761748 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:17.761719    4492 scope.go:122] "RemoveContainer" containerID="addf4172f339115eb6e05e525e1b8f4ea43f6a1a95e844fa2855eba3d091763d"
Nov 02 14:56:18.886403 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:18.886336    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:20.735863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:20.735833    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:23.173998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:23.173955    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:25.495809 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:25.495773    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:27.330914 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:27.330808    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:29.812744 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:29.812711    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:32.108385 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:32.108325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:33.931621 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:33.931585    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:34.070767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:34.070734    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr_ec4b89e1-4ca8-4008-939d-6201d69bac90/payload-processing/0.log"
Nov 02 14:56:36.314239 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:36.314202    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:38.680518 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:38.680464    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:40.528791 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:40.528754    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:42.369195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:42.369159    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:42.924392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:42.924346    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:45.188878 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:45.188843    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:46.517714 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:56:46.517678    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:56:46.787722 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:46.787634    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:46.902937 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:46.902903    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:47.024761 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:47.024729    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:48.089638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:48.089604    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:49.425581 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:49.425543    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:51.775848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:51.775811    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:53.144905 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:53.144856    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:53.662967 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:53.662932    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:56.016767 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:56.016730    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:58.402025 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:56:58.401984    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:00.231437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:00.231403    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:02.620858 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:02.620822    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:04.988536 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:04.988500    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:06.831566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:06.831531    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:09.151502 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:09.151464    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:11.399416 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:11.399381    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:13.219044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:13.218992    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:15.620243 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:15.620209    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:16.516647 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:57:16.516612    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:57:17.062542 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:17.062505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:17.790107 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:17.790062    4492 scope.go:122] "RemoveContainer" containerID="6cde04455836917eaf28622185ff01074b920e6d847a5f775156e42123feafce"
Nov 02 14:57:17.800219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:17.800195    4492 scope.go:122] "RemoveContainer" containerID="b4ef6c79ea41645da29fc203fb1406a0c5e123cc05e48f414a40a3c71b79dc88"
Nov 02 14:57:17.809208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:17.809188    4492 scope.go:122] "RemoveContainer" containerID="7074700e19fd75edd16acbbf26a0b0f2b32c45e6d86f35eaf2a983fd52e46c54"
Nov 02 14:57:17.988630 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:17.988588    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:19.830539 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:19.830505    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:22.228219 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:22.228184    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:24.492623 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:24.492587    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:26.422831 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:26.422797    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:28.835334 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:28.835299    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:29.713689 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:29.713651    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:31.208228 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:31.208186    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:31.318419 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:31.318386    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:33.226840 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:33.226780    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:35.500459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:35.500425    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:37.329399 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:37.329366    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:37.836327 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:37.836292    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:39.688789 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:39.688752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:41.522113 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:41.522073    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:43.652344 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.652309    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:43.760486 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.760449    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"]
Nov 02 14:57:43.760732 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.760707    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr" podUID="ec4b89e1-4ca8-4008-939d-6201d69bac90" containerName="payload-processing" containerID="cri-o://a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7" gracePeriod=30
Nov 02 14:57:43.783170 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.783139    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"]
Nov 02 14:57:43.783654 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.783430    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97" podUID="4cd228fb-67ba-452f-af5e-6aab6a15b2a4" containerName="payload-pre-processing" containerID="cri-o://124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c" gracePeriod=30
Nov 02 14:57:43.849505 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.849478    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:43.965208 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:43.965121    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:44.139734 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.134636    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:57:44.156469 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.156398    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:57:44.221779 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.221746    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume\") pod \"ec4b89e1-4ca8-4008-939d-6201d69bac90\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") "
Nov 02 14:57:44.221987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.221868    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume\") pod \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") "
Nov 02 14:57:44.221987 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.221933    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4\" (UniqueName: \"kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4\") pod \"ec4b89e1-4ca8-4008-939d-6201d69bac90\" (UID: \"ec4b89e1-4ca8-4008-939d-6201d69bac90\") "
Nov 02 14:57:44.222126 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.221999    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch\" (UniqueName: \"kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch\") pod \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\" (UID: \"4cd228fb-67ba-452f-af5e-6aab6a15b2a4\") "
Nov 02 14:57:44.222438 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.222236    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume" pod "ec4b89e1-4ca8-4008-939d-6201d69bac90" (UID: "ec4b89e1-4ca8-4008-939d-6201d69bac90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:44.222563 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.222454    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4b89e1-4ca8-4008-939d-6201d69bac90-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.223000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.222919    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:44.223100 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.223009    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume" pod "4cd228fb-67ba-452f-af5e-6aab6a15b2a4" (UID: "4cd228fb-67ba-452f-af5e-6aab6a15b2a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:44.224893 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.224869    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4" pod "ec4b89e1-4ca8-4008-939d-6201d69bac90" (UID: "ec4b89e1-4ca8-4008-939d-6201d69bac90"). InnerVolumeSpecName "kube-api-access-s9jl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:44.225178 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.225149    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch" pod "4cd228fb-67ba-452f-af5e-6aab6a15b2a4" (UID: "4cd228fb-67ba-452f-af5e-6aab6a15b2a4"). InnerVolumeSpecName "kube-api-access-4ntch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:44.323394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.323343    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.323394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.323391    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jl4\" (UniqueName: \"kubernetes.io/projected/ec4b89e1-4ca8-4008-939d-6201d69bac90-kube-api-access-s9jl4\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.323394 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.323401    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntch\" (UniqueName: \"kubernetes.io/projected/4cd228fb-67ba-452f-af5e-6aab6a15b2a4-kube-api-access-4ntch\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.383189 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.383152    4492 generic.go:356] "Generic (PLEG): container finished" podID="ec4b89e1-4ca8-4008-939d-6201d69bac90" containerID="a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7" exitCode=0
Nov 02 14:57:44.383407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.383204    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"
Nov 02 14:57:44.383407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.383235    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr" event={"ID":"ec4b89e1-4ca8-4008-939d-6201d69bac90","Type":"ContainerDied","Data":"a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7"}
Nov 02 14:57:44.383407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.383294    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr" event={"ID":"ec4b89e1-4ca8-4008-939d-6201d69bac90","Type":"ContainerDied","Data":"0a26ed4f6280ad27fe91aa353f6aae773bc32bedfd7c95d3c91baa5fd5064f4a"}
Nov 02 14:57:44.383407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.383320    4492 scope.go:122] "RemoveContainer" containerID="a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7"
Nov 02 14:57:44.385168 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.385141    4492 generic.go:356] "Generic (PLEG): container finished" podID="4cd228fb-67ba-452f-af5e-6aab6a15b2a4" containerID="124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c" exitCode=0
Nov 02 14:57:44.385308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.385226    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97" event={"ID":"4cd228fb-67ba-452f-af5e-6aab6a15b2a4","Type":"ContainerDied","Data":"124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c"}
Nov 02 14:57:44.385308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.385235    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"
Nov 02 14:57:44.385308 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.385272    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97" event={"ID":"4cd228fb-67ba-452f-af5e-6aab6a15b2a4","Type":"ContainerDied","Data":"cadef7a37156433562c4da1ef54b3a06eb7e35190c8fdeb07c8ea9365ff23d7d"}
Nov 02 14:57:44.403028 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.402986    4492 scope.go:122] "RemoveContainer" containerID="a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7"
Nov 02 14:57:44.403613 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:57:44.403585    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7\": container with ID starting with a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7 not found: ID does not exist" containerID="a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7"
Nov 02 14:57:44.403724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.403623    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7"} err="failed to get container status \"a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7\": rpc error: code = NotFound desc = could not find container \"a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7\": container with ID starting with a7c303beb7547b4f367370035d635b470306e88fc941985638e35bcd85de78d7 not found: ID does not exist"
Nov 02 14:57:44.403724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.403649    4492 scope.go:122] "RemoveContainer" containerID="124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c"
Nov 02 14:57:44.414867 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.414840    4492 scope.go:122] "RemoveContainer" containerID="124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c"
Nov 02 14:57:44.415196 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:57:44.415174    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c\": container with ID starting with 124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c not found: ID does not exist" containerID="124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c"
Nov 02 14:57:44.415269 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.415211    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c"} err="failed to get container status \"124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c\": rpc error: code = NotFound desc = could not find container \"124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c\": container with ID starting with 124ce473a83e7f22b08dfc1089a776edc942140528fdcdb8f2712f5ff4dfd77c not found: ID does not exist"
Nov 02 14:57:44.438802 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.438771    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"]
Nov 02 14:57:44.444437 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.444411    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b21aae-5879fdf959-cms97"]
Nov 02 14:57:44.474700 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.474669    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"]
Nov 02 14:57:44.478256 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.478225    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:44.479383 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:44.479333    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b21aae-6bc4bd8db-85wvr"]
Nov 02 14:57:45.583046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:45.583007    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd228fb-67ba-452f-af5e-6aab6a15b2a4" path="/var/lib/kubelet/pods/4cd228fb-67ba-452f-af5e-6aab6a15b2a4/volumes"
Nov 02 14:57:45.583638 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:45.583621    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4b89e1-4ca8-4008-939d-6201d69bac90" path="/var/lib/kubelet/pods/ec4b89e1-4ca8-4008-939d-6201d69bac90/volumes"
Nov 02 14:57:46.349694 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:46.349660    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:46.516612 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:57:46.516578    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:57:48.715085 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:48.715023    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:50.624649 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:50.624611    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:50.957564 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:50.957533    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:51.071884 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:51.071848    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:51.740441 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:51.740403    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:52.241000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:52.240965    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"]
Nov 02 14:57:52.241222 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:52.241201    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy" containerID="cri-o://232ebd3356aaa8783dd6a489891353b5780466d0193a13050dbc11fff1c92c33" gracePeriod=30
Nov 02 14:57:52.285467 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:52.285262    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:52.505471 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:52.505375    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:53.195497 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:53.195462    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:55.133468 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:55.133438    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:57.131742 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.131657    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:57.493743 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.493707    4492 generic.go:356] "Generic (PLEG): container finished" podID="336cc98d-960d-4b59-9a21-398e177f1924" containerID="232ebd3356aaa8783dd6a489891353b5780466d0193a13050dbc11fff1c92c33" exitCode=0
Nov 02 14:57:57.493902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.493774    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" event={"ID":"336cc98d-960d-4b59-9a21-398e177f1924","Type":"ContainerDied","Data":"232ebd3356aaa8783dd6a489891353b5780466d0193a13050dbc11fff1c92c33"}
Nov 02 14:57:57.493902 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.493879    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh" event={"ID":"336cc98d-960d-4b59-9a21-398e177f1924","Type":"ContainerDied","Data":"9434b0990ffa36c8361f47b611a20f07c8cde4bb86341d2230df15ea72aef9e7"}
Nov 02 14:57:57.494000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.493904    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9434b0990ffa36c8361f47b611a20f07c8cde4bb86341d2230df15ea72aef9e7"
Nov 02 14:57:57.494000 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.493918    4492 scope.go:122] "RemoveContainer" containerID="e41fb4258a0da068938e43a29621f9365894d900cdf690f549fce190cbc8898f"
Nov 02 14:57:57.502246 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.502218    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:57.508280 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.508262    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:57:57.592260 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592226    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.592434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592276    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.592434 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592332    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.592577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592434    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.592577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592501    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.592577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592536    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.592577 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592553    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.592798 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592592    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.592883 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592849    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.593027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.592959    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.593027 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593005    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.593141 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593066    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf\") pod \"336cc98d-960d-4b59-9a21-398e177f1924\" (UID: \"336cc98d-960d-4b59-9a21-398e177f1924\") "
Nov 02 14:57:57.593422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593397    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.593422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593424    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.593422 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593437    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.593760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593676    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.593760 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.593715    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:57.595371 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.595326    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.595490 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.595437    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:57.596099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.596063    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:57.596099 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.596069    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf" pod "336cc98d-960d-4b59-9a21-398e177f1924" (UID: "336cc98d-960d-4b59-9a21-398e177f1924"). InnerVolumeSpecName "kube-api-access-m9bqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:57.694500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.694454    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.694500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.694495    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cc98d-960d-4b59-9a21-398e177f1924-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.694500 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.694511    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cc98d-960d-4b59-9a21-398e177f1924-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.694781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.694525    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bqf\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-kube-api-access-m9bqf\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.694781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.694537    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cc98d-960d-4b59-9a21-398e177f1924-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.694781 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:57.694551    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cc98d-960d-4b59-9a21-398e177f1924-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:57:58.366714 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:58.366683    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:58.501695 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:58.501663    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"
Nov 02 14:57:58.585924 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:58.585888    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"]
Nov 02 14:57:58.591131 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:58.591100    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b21aae-openshift-default-7f5b5c687f-b44jh"]
Nov 02 14:57:59.314724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:59.314664    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:59.580302 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:57:59.580217    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336cc98d-960d-4b59-9a21-398e177f1924" path="/var/lib/kubelet/pods/336cc98d-960d-4b59-9a21-398e177f1924/volumes"
Nov 02 14:58:01.735724 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:01.735685    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:03.659627 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:03.659588    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:05.245417 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:05.245384    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:06.085578 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:06.085540    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:07.553544 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.553510    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"]
Nov 02 14:58:07.554024 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.553737    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn" podUID="4b7b73da-1320-4c9e-8483-576d999290d7" containerName="maas-api" containerID="cri-o://61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec" gracePeriod=30
Nov 02 14:58:07.617227 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.617196    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:07.823784 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.823676    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:07.850193 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.850164    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:58:07.901091 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.901053    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv\" (UniqueName: \"kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv\") pod \"4b7b73da-1320-4c9e-8483-576d999290d7\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") "
Nov 02 14:58:07.901267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.901164    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs\") pod \"4b7b73da-1320-4c9e-8483-576d999290d7\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") "
Nov 02 14:58:07.901267 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.901198    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls\") pod \"4b7b73da-1320-4c9e-8483-576d999290d7\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") "
Nov 02 14:58:07.901392 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.901266    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca\") pod \"4b7b73da-1320-4c9e-8483-576d999290d7\" (UID: \"4b7b73da-1320-4c9e-8483-576d999290d7\") "
Nov 02 14:58:07.902083 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.902040    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca" pod "4b7b73da-1320-4c9e-8483-576d999290d7" (UID: "4b7b73da-1320-4c9e-8483-576d999290d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:07.903613 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.903580    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls" pod "4b7b73da-1320-4c9e-8483-576d999290d7" (UID: "4b7b73da-1320-4c9e-8483-576d999290d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:07.904044 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.904023    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs" pod "4b7b73da-1320-4c9e-8483-576d999290d7" (UID: "4b7b73da-1320-4c9e-8483-576d999290d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:07.904097 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.904063    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv" pod "4b7b73da-1320-4c9e-8483-576d999290d7" (UID: "4b7b73da-1320-4c9e-8483-576d999290d7"). InnerVolumeSpecName "kube-api-access-2mqdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:07.931745 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:07.931715    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:08.003143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.003107    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mqdv\" (UniqueName: \"kubernetes.io/projected/4b7b73da-1320-4c9e-8483-576d999290d7-kube-api-access-2mqdv\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:58:08.003143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.003135    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:58:08.003143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.003143    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b7b73da-1320-4c9e-8483-576d999290d7-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:58:08.003143 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.003152    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7b73da-1320-4c9e-8483-576d999290d7-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:58:08.560488 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.560450    4492 generic.go:356] "Generic (PLEG): container finished" podID="4b7b73da-1320-4c9e-8483-576d999290d7" containerID="61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec" exitCode=0
Nov 02 14:58:08.560966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.560498    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"
Nov 02 14:58:08.560966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.560537    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn" event={"ID":"4b7b73da-1320-4c9e-8483-576d999290d7","Type":"ContainerDied","Data":"61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec"}
Nov 02 14:58:08.560966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.560590    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn" event={"ID":"4b7b73da-1320-4c9e-8483-576d999290d7","Type":"ContainerDied","Data":"9b1f78087328a8274c8a065c25880bec37524f5e1efd6cd5839482cba376c4bb"}
Nov 02 14:58:08.560966 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.560614    4492 scope.go:122] "RemoveContainer" containerID="61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec"
Nov 02 14:58:08.571585 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.571559    4492 scope.go:122] "RemoveContainer" containerID="61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec"
Nov 02 14:58:08.571905 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:58:08.571885    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec\": container with ID starting with 61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec not found: ID does not exist" containerID="61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec"
Nov 02 14:58:08.571945 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.571915    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec"} err="failed to get container status \"61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec\": rpc error: code = NotFound desc = could not find container \"61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec\": container with ID starting with 61cdb976e170364d68bfb7d6aff438bf47de00bb7ad708139520c9def1bf46ec not found: ID does not exist"
Nov 02 14:58:08.607688 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.607651    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"]
Nov 02 14:58:08.617397 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:08.617348    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be6018-69695589c8-d8swn"]
Nov 02 14:58:09.579454 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:09.579409    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7b73da-1320-4c9e-8483-576d999290d7" path="/var/lib/kubelet/pods/4b7b73da-1320-4c9e-8483-576d999290d7/volumes"
Nov 02 14:58:11.519479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:11.519439    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:11.629771 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:11.629725    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:12.242463 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:12.242426    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:12.663836 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:12.663748    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:14.171530 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.171485    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"]
Nov 02 14:58:14.172407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172381    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4cd228fb-67ba-452f-af5e-6aab6a15b2a4" containerName="payload-pre-processing"
Nov 02 14:58:14.172407 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172404    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd228fb-67ba-452f-af5e-6aab6a15b2a4" containerName="payload-pre-processing"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172442    4492 cpu_manager.go:414] "containerMap: removing container" podUID="ec4b89e1-4ca8-4008-939d-6201d69bac90" containerName="payload-processing"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172452    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4b89e1-4ca8-4008-939d-6201d69bac90" containerName="payload-processing"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172482    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4b7b73da-1320-4c9e-8483-576d999290d7" containerName="maas-api"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172488    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7b73da-1320-4c9e-8483-576d999290d7" containerName="maas-api"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172516    4492 cpu_manager.go:414] "containerMap: removing container" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172522    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172542    4492 cpu_manager.go:414] "containerMap: removing container" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy"
Nov 02 14:58:14.172615 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172554    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy"
Nov 02 14:58:14.173001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172726    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd228fb-67ba-452f-af5e-6aab6a15b2a4" containerName="payload-pre-processing"
Nov 02 14:58:14.173001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172749    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4b89e1-4ca8-4008-939d-6201d69bac90" containerName="payload-processing"
Nov 02 14:58:14.173001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172759    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7b73da-1320-4c9e-8483-576d999290d7" containerName="maas-api"
Nov 02 14:58:14.173001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172774    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy"
Nov 02 14:58:14.173001 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.172789    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="336cc98d-960d-4b59-9a21-398e177f1924" containerName="istio-proxy"
Nov 02 14:58:14.178412 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.178391    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.194602 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.194568    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"]
Nov 02 14:58:14.200670 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.200638    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 14:58:14.200848 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.200714    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-v4g45\""
Nov 02 14:58:14.277183 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.277140    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6fh\" (UniqueName: \"kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh\") pod \"payload-pre-processing-574bdd7c8c-s7zkh\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") " pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.277408 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.277197    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume\") pod \"payload-pre-processing-574bdd7c8c-s7zkh\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") " pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.378038 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.377999    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6fh\" (UniqueName: \"kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh\") pod \"payload-pre-processing-574bdd7c8c-s7zkh\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") " pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.378249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.378052    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume\") pod \"payload-pre-processing-574bdd7c8c-s7zkh\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") " pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.378860 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.378833    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume\") pod \"payload-pre-processing-574bdd7c8c-s7zkh\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") " pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.399946 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.399908    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6fh\" (UniqueName: \"kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh\") pod \"payload-pre-processing-574bdd7c8c-s7zkh\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") " pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.492614 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.492583    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:14.793459 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:14.793380    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"]
Nov 02 14:58:14.797519 ip-10-0-129-115 kubenswrapper[4492]: W1102 14:58:14.797487    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0618a9_14b4_41a6_803f_b3e332502801.slice/crio-8c38d20bc84bdc08c711977ef4f3c8fe322074d8c0d3316fc91c3db8d9f2ddb8 WatchSource:0}: Error finding container 8c38d20bc84bdc08c711977ef4f3c8fe322074d8c0d3316fc91c3db8d9f2ddb8: Status 404 returned error can't find the container with id 8c38d20bc84bdc08c711977ef4f3c8fe322074d8c0d3316fc91c3db8d9f2ddb8
Nov 02 14:58:15.615629 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:15.615584    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh" event={"ID":"2f0618a9-14b4-41a6-803f-b3e332502801","Type":"ContainerStarted","Data":"d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6"}
Nov 02 14:58:15.616046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:15.615647    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh" event={"ID":"2f0618a9-14b4-41a6-803f-b3e332502801","Type":"ContainerStarted","Data":"8c38d20bc84bdc08c711977ef4f3c8fe322074d8c0d3316fc91c3db8d9f2ddb8"}
Nov 02 14:58:15.616046 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:15.615676    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:15.655195 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:15.655143    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh" podStartSLOduration=1.655129359 podStartE2EDuration="1.655129359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:15.652429938 +0000 UTC m=+3660.196145533" watchObservedRunningTime="2026-11-02 14:58:15.655129359 +0000 UTC m=+3660.198844988"
Nov 02 14:58:16.517624 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:58:16.517587    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:58:17.200174 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:17.200142    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:26.624450 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:26.624418    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:58:30.035326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:30.035271    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"]
Nov 02 14:58:30.044507 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:30.044469    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bsl92"]
Nov 02 14:58:31.579812 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:31.579773    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ff27ac-41a6-4fc9-b74a-45e717e744d1" path="/var/lib/kubelet/pods/f7ff27ac-41a6-4fc9-b74a-45e717e744d1/volumes"
Nov 02 14:58:32.143074 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:32.143042    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:46.516864 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:58:46.516825    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:58:48.034253 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:48.034211    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"]
Nov 02 14:58:48.043776 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:48.043742    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-bvr5z"]
Nov 02 14:58:49.580479 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:49.580444    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd21e720-a86b-4f41-a5c3-6661ff55113c" path="/var/lib/kubelet/pods/fd21e720-a86b-4f41-a5c3-6661ff55113c/volumes"
Nov 02 14:58:52.615082 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:52.615029    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:54.952863 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:54.952827    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"]
Nov 02 14:58:56.617566 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:56.617528    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:56.730645 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:56.730614    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:57.319652 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:57.319621    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:57.507400 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:57.507339    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:58.180056 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:58.180021    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:59.953673 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:58:59.953632    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh" podUID="2f0618a9-14b4-41a6-803f-b3e332502801" containerName="payload-pre-processing" containerID="cri-o://d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6" gracePeriod=25
Nov 02 14:59:00.321561 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.321198    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:59:00.488032 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.487978    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh\" (UniqueName: \"kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh\") pod \"2f0618a9-14b4-41a6-803f-b3e332502801\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") "
Nov 02 14:59:00.488255 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.488048    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume\") pod \"2f0618a9-14b4-41a6-803f-b3e332502801\" (UID: \"2f0618a9-14b4-41a6-803f-b3e332502801\") "
Nov 02 14:59:00.488908 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.488876    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume" pod "2f0618a9-14b4-41a6-803f-b3e332502801" (UID: "2f0618a9-14b4-41a6-803f-b3e332502801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:00.492145 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.492114    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh" pod "2f0618a9-14b4-41a6-803f-b3e332502801" (UID: "2f0618a9-14b4-41a6-803f-b3e332502801"). InnerVolumeSpecName "kube-api-access-gw6fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:00.590898 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.590804    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6fh\" (UniqueName: \"kubernetes.io/projected/2f0618a9-14b4-41a6-803f-b3e332502801-kube-api-access-gw6fh\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:59:00.591396 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.591371    4492 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0618a9-14b4-41a6-803f-b3e332502801-plugins-config-volume\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 14:59:00.870850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.870744    4492 generic.go:356] "Generic (PLEG): container finished" podID="2f0618a9-14b4-41a6-803f-b3e332502801" containerID="d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6" exitCode=0
Nov 02 14:59:00.870850 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.870800    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh" event={"ID":"2f0618a9-14b4-41a6-803f-b3e332502801","Type":"ContainerDied","Data":"d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6"}
Nov 02 14:59:00.871130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.870851    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh" event={"ID":"2f0618a9-14b4-41a6-803f-b3e332502801","Type":"ContainerDied","Data":"8c38d20bc84bdc08c711977ef4f3c8fe322074d8c0d3316fc91c3db8d9f2ddb8"}
Nov 02 14:59:00.871130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.870882    4492 scope.go:122] "RemoveContainer" containerID="d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6"
Nov 02 14:59:00.871130 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.871058    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"
Nov 02 14:59:00.885719 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.885693    4492 scope.go:122] "RemoveContainer" containerID="d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6"
Nov 02 14:59:00.886289 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:59:00.886262    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6\": container with ID starting with d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6 not found: ID does not exist" containerID="d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6"
Nov 02 14:59:00.886768 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.886302    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6"} err="failed to get container status \"d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6\": rpc error: code = NotFound desc = could not find container \"d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6\": container with ID starting with d820f9129a37423fd5b0ff10ebcf078dda0776d55a0cb7ec9e6fd948630f0cc6 not found: ID does not exist"
Nov 02 14:59:00.920386 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.920330    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"]
Nov 02 14:59:00.935998 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:00.935960    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-574bdd7c8c-s7zkh"]
Nov 02 14:59:01.591080 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:01.591043    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0618a9-14b4-41a6-803f-b3e332502801" path="/var/lib/kubelet/pods/2f0618a9-14b4-41a6-803f-b3e332502801/volumes"
Nov 02 14:59:09.037567 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:09.037534    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"]
Nov 02 14:59:09.046868 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:09.046841    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25b36506-mphb2"]
Nov 02 14:59:09.579326 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:09.579293    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af26be2-1d78-4b96-9f1f-d4ef315335b0" path="/var/lib/kubelet/pods/5af26be2-1d78-4b96-9f1f-d4ef315335b0/volumes"
Nov 02 14:59:16.517338 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:59:16.517301    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:59:17.335249 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:17.335214    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:32.269726 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:32.269692    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:37.768277 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:37.768243    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:46.517265 ip-10-0-129-115 kubenswrapper[4492]: E1102 14:59:46.517227    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 14:59:52.746483 ip-10-0-129-115 kubenswrapper[4492]: I1102 14:59:52.746445    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:00.156249 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.156213    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"]
Nov 02 15:00:00.156702 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.156670    4492 cpu_manager.go:414] "containerMap: removing container" podUID="2f0618a9-14b4-41a6-803f-b3e332502801" containerName="payload-pre-processing"
Nov 02 15:00:00.156702 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.156684    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0618a9-14b4-41a6-803f-b3e332502801" containerName="payload-pre-processing"
Nov 02 15:00:00.156778 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.156772    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0618a9-14b4-41a6-803f-b3e332502801" containerName="payload-pre-processing"
Nov 02 15:00:00.160003 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.159985    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.178213 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.178183    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:00:00.178387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.178242    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:00:00.188155 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.188124    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:00:00.191203 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.191178    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"]
Nov 02 15:00:00.256028 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.255997    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893860-6m6tb\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.256225 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.256050    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9zp\" (UniqueName: \"kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp\") pod \"maas-api-key-cleanup-29893860-6m6tb\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.356926 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.356884    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893860-6m6tb\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.357137 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.356965    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9zp\" (UniqueName: \"kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp\") pod \"maas-api-key-cleanup-29893860-6m6tb\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.357669 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.357647    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893860-6m6tb\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.378805 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.378773    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9zp\" (UniqueName: \"kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp\") pod \"maas-api-key-cleanup-29893860-6m6tb\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.472902 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.472869    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:00.771920 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:00.771844    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"]
Nov 02 15:00:00.775655 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:00:00.775627    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91825ed1_c020_4fde_af5e_c7864b6d716e.slice/crio-4b534a2d88b594bb809289b39ab3cfbf9c263864ae9b475185304285c91e8e17 WatchSource:0}: Error finding container 4b534a2d88b594bb809289b39ab3cfbf9c263864ae9b475185304285c91e8e17: Status 404 returned error can't find the container with id 4b534a2d88b594bb809289b39ab3cfbf9c263864ae9b475185304285c91e8e17
Nov 02 15:00:01.255620 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:01.255586    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerStarted","Data":"8a3e196c2dc076b6c88890f8f275b91baa46dfd83c8e8f66511aaab65e9672d1"}
Nov 02 15:00:01.256043 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:01.255640    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerStarted","Data":"4b534a2d88b594bb809289b39ab3cfbf9c263864ae9b475185304285c91e8e17"}
Nov 02 15:00:01.293403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:01.293266    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" podStartSLOduration=1.293252007 podStartE2EDuration="1.293252007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:01.29173843 +0000 UTC m=+3765.835454025" watchObservedRunningTime="2026-11-02 15:00:01.293252007 +0000 UTC m=+3765.836967600"
Nov 02 15:00:02.262722 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:02.262680    4492 generic.go:356] "Generic (PLEG): container finished" podID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerID="8a3e196c2dc076b6c88890f8f275b91baa46dfd83c8e8f66511aaab65e9672d1" exitCode=60
Nov 02 15:00:02.263149 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:02.262769    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerDied","Data":"8a3e196c2dc076b6c88890f8f275b91baa46dfd83c8e8f66511aaab65e9672d1"}
Nov 02 15:00:02.263149 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:02.262991    4492 scope.go:122] "RemoveContainer" containerID="8a3e196c2dc076b6c88890f8f275b91baa46dfd83c8e8f66511aaab65e9672d1"
Nov 02 15:00:03.271147 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:03.271117    4492 generic.go:356] "Generic (PLEG): container finished" podID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerID="5f1948d442ac834c5e565995ccfb4eede5b534c7358de5a3967dbaf52ff29900" exitCode=60
Nov 02 15:00:03.271775 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:03.271199    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerDied","Data":"5f1948d442ac834c5e565995ccfb4eede5b534c7358de5a3967dbaf52ff29900"}
Nov 02 15:00:03.271775 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:03.271271    4492 scope.go:122] "RemoveContainer" containerID="8a3e196c2dc076b6c88890f8f275b91baa46dfd83c8e8f66511aaab65e9672d1"
Nov 02 15:00:03.271775 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:03.271544    4492 scope.go:122] "RemoveContainer" containerID="5f1948d442ac834c5e565995ccfb4eede5b534c7358de5a3967dbaf52ff29900"
Nov 02 15:00:03.271920 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:00:03.271807    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893860-6m6tb_odh-ai-gateway-infra(91825ed1-c020-4fde-af5e-c7864b6d716e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e"
Nov 02 15:00:04.278521 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:04.278487    4492 scope.go:122] "RemoveContainer" containerID="5f1948d442ac834c5e565995ccfb4eede5b534c7358de5a3967dbaf52ff29900"
Nov 02 15:00:04.278948 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:00:04.278744    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893860-6m6tb_odh-ai-gateway-infra(91825ed1-c020-4fde-af5e-c7864b6d716e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e"
Nov 02 15:00:14.574395 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:14.574338    4492 scope.go:122] "RemoveContainer" containerID="5f1948d442ac834c5e565995ccfb4eede5b534c7358de5a3967dbaf52ff29900"
Nov 02 15:00:15.341295 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:15.341255    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerStarted","Data":"1db4e874a460cd2ac2e17ae88f431f5d64570515ba935a785e6eb97dd594108a"}
Nov 02 15:00:16.348519 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:16.348485    4492 generic.go:356] "Generic (PLEG): container finished" podID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerID="1db4e874a460cd2ac2e17ae88f431f5d64570515ba935a785e6eb97dd594108a" exitCode=60
Nov 02 15:00:16.349022 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:16.348557    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerDied","Data":"1db4e874a460cd2ac2e17ae88f431f5d64570515ba935a785e6eb97dd594108a"}
Nov 02 15:00:16.349022 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:16.348620    4492 scope.go:122] "RemoveContainer" containerID="5f1948d442ac834c5e565995ccfb4eede5b534c7358de5a3967dbaf52ff29900"
Nov 02 15:00:16.349022 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:16.348973    4492 scope.go:122] "RemoveContainer" containerID="1db4e874a460cd2ac2e17ae88f431f5d64570515ba935a785e6eb97dd594108a"
Nov 02 15:00:16.349309 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:00:16.349286    4492 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893860-6m6tb_odh-ai-gateway-infra(91825ed1-c020-4fde-af5e-c7864b6d716e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e"
Nov 02 15:00:16.400160 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:16.400123    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"]
Nov 02 15:00:16.517273 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:00:16.517236    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:00:17.502271 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.502244    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:17.529516 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.529486    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp\" (UniqueName: \"kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp\") pod \"91825ed1-c020-4fde-af5e-c7864b6d716e\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") "
Nov 02 15:00:17.529668 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.529582    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca\") pod \"91825ed1-c020-4fde-af5e-c7864b6d716e\" (UID: \"91825ed1-c020-4fde-af5e-c7864b6d716e\") "
Nov 02 15:00:17.529949 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.529920    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca" pod "91825ed1-c020-4fde-af5e-c7864b6d716e" (UID: "91825ed1-c020-4fde-af5e-c7864b6d716e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:00:17.531794 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.531775    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp" pod "91825ed1-c020-4fde-af5e-c7864b6d716e" (UID: "91825ed1-c020-4fde-af5e-c7864b6d716e"). InnerVolumeSpecName "kube-api-access-5l9zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:17.568282 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.568244    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:17.631865 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.631778    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91825ed1-c020-4fde-af5e-c7864b6d716e-maas-api-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:00:17.631865 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.631816    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9zp\" (UniqueName: \"kubernetes.io/projected/91825ed1-c020-4fde-af5e-c7864b6d716e-kube-api-access-5l9zp\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:00:17.879953 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.879921    4492 scope.go:122] "RemoveContainer" containerID="f10d7b811542bd28918eb2db25383f6381803e425d2c2c8884e8a58945b5ed6b"
Nov 02 15:00:17.889444 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.889415    4492 scope.go:122] "RemoveContainer" containerID="a2b11beb1aa3cfaf3c470776dd9a3dfe543bdb53ca5436b093d04620df977db2"
Nov 02 15:00:17.898235 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:17.898214    4492 scope.go:122] "RemoveContainer" containerID="c41c177924f669f3efc7dd3e4b927e31fd2ec7aa107d1a82383c562b8fbe3311"
Nov 02 15:00:18.363492 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:18.363459    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"
Nov 02 15:00:18.363683 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:18.363489    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb" event={"ID":"91825ed1-c020-4fde-af5e-c7864b6d716e","Type":"ContainerDied","Data":"4b534a2d88b594bb809289b39ab3cfbf9c263864ae9b475185304285c91e8e17"}
Nov 02 15:00:18.363683 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:18.363583    4492 scope.go:122] "RemoveContainer" containerID="1db4e874a460cd2ac2e17ae88f431f5d64570515ba935a785e6eb97dd594108a"
Nov 02 15:00:18.411022 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:18.410984    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"]
Nov 02 15:00:18.420712 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:18.420682    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-6m6tb"]
Nov 02 15:00:19.581183 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:19.581150    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" path="/var/lib/kubelet/pods/91825ed1-c020-4fde-af5e-c7864b6d716e/volumes"
Nov 02 15:00:30.634678 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:30.634629    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:30.802467 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:30.802429    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:32.412974 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:32.412937    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:35.399156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399112    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"]
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399586    4492 cpu_manager.go:414] "containerMap: removing container" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399601    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399611    4492 cpu_manager.go:414] "containerMap: removing container" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399616    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399629    4492 cpu_manager.go:414] "containerMap: removing container" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399634    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399725    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399734    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.399783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.399741    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="91825ed1-c020-4fde-af5e-c7864b6d716e" containerName="cleanup"
Nov 02 15:00:35.402762 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.402745    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.427467 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.427437    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"]
Nov 02 15:00:35.434199 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.434164    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:00:35.434442 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.434200    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:00:35.434442 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.434270    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:00:35.434442 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.434285    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:00:35.522281 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.522233    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.522494 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.522296    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.522494 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.522331    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmld6\" (UniqueName: \"kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.623835 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.623791    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.623835 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.623836    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.624082 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.623858    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmld6\" (UniqueName: \"kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.624604 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.624583    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.626530 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.626512    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.645346 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.645315    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmld6\" (UniqueName: \"kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6\") pod \"maas-api-revoke-sub-e2547622-js5fh\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:35.716487 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:35.716454    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:36.020158 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:36.020075    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"]
Nov 02 15:00:36.024014 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:00:36.023979    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83732677_acea_47fa_9b17_486c2f33efc9.slice/crio-228ee5ef78e7f4706c10486460717b5364c1099aaef9ddee32a3ef6d5ba7aacf WatchSource:0}: Error finding container 228ee5ef78e7f4706c10486460717b5364c1099aaef9ddee32a3ef6d5ba7aacf: Status 404 returned error can't find the container with id 228ee5ef78e7f4706c10486460717b5364c1099aaef9ddee32a3ef6d5ba7aacf
Nov 02 15:00:36.468898 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:36.468857    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh" event={"ID":"83732677-acea-47fa-9b17-486c2f33efc9","Type":"ContainerStarted","Data":"7d76ff7ebc6812ff04c59f1f3e96ee34aa3827933aad1edc7ef35da66ec10df9"}
Nov 02 15:00:36.469330 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:36.468917    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh" event={"ID":"83732677-acea-47fa-9b17-486c2f33efc9","Type":"ContainerStarted","Data":"228ee5ef78e7f4706c10486460717b5364c1099aaef9ddee32a3ef6d5ba7aacf"}
Nov 02 15:00:36.515283 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:36.515230    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh" podStartSLOduration=1.5152140699999999 podStartE2EDuration="1.51521407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:36.512548317 +0000 UTC m=+3801.056263926" watchObservedRunningTime="2026-11-02 15:00:36.51521407 +0000 UTC m=+3801.058929661"
Nov 02 15:00:37.476460 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:37.476429    4492 generic.go:356] "Generic (PLEG): container finished" podID="83732677-acea-47fa-9b17-486c2f33efc9" containerID="7d76ff7ebc6812ff04c59f1f3e96ee34aa3827933aad1edc7ef35da66ec10df9" exitCode=0
Nov 02 15:00:37.476877 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:37.476518    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh" event={"ID":"83732677-acea-47fa-9b17-486c2f33efc9","Type":"ContainerDied","Data":"7d76ff7ebc6812ff04c59f1f3e96ee34aa3827933aad1edc7ef35da66ec10df9"}
Nov 02 15:00:38.630090 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.630062    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:38.756496 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.756408    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6\" (UniqueName: \"kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6\") pod \"83732677-acea-47fa-9b17-486c2f33efc9\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") "
Nov 02 15:00:38.756656 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.756511    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca\") pod \"83732677-acea-47fa-9b17-486c2f33efc9\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") "
Nov 02 15:00:38.756656 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.756573    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca\") pod \"83732677-acea-47fa-9b17-486c2f33efc9\" (UID: \"83732677-acea-47fa-9b17-486c2f33efc9\") "
Nov 02 15:00:38.756958 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.756929    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca" pod "83732677-acea-47fa-9b17-486c2f33efc9" (UID: "83732677-acea-47fa-9b17-486c2f33efc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:00:38.758672 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.758644    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6" pod "83732677-acea-47fa-9b17-486c2f33efc9" (UID: "83732677-acea-47fa-9b17-486c2f33efc9"). InnerVolumeSpecName "kube-api-access-vmld6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:38.758773 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.758709    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca" pod "83732677-acea-47fa-9b17-486c2f33efc9" (UID: "83732677-acea-47fa-9b17-486c2f33efc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:00:38.857838 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.857798    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmld6\" (UniqueName: \"kubernetes.io/projected/83732677-acea-47fa-9b17-486c2f33efc9-kube-api-access-vmld6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:00:38.857838 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.857829    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:00:38.857838 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:38.857849    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83732677-acea-47fa-9b17-486c2f33efc9-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:00:39.490096 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:39.490063    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"
Nov 02 15:00:39.490302 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:39.490110    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh" event={"ID":"83732677-acea-47fa-9b17-486c2f33efc9","Type":"ContainerDied","Data":"228ee5ef78e7f4706c10486460717b5364c1099aaef9ddee32a3ef6d5ba7aacf"}
Nov 02 15:00:39.490302 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:39.490198    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228ee5ef78e7f4706c10486460717b5364c1099aaef9ddee32a3ef6d5ba7aacf"
Nov 02 15:00:46.250429 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:46.250390    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:46.372302 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:46.372271    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:46.547015 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:00:46.546923    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:00:46.650509 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:46.650473    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:49.960509 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:49.960469    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:00:49.961103 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:49.961068    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:00:49.983379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:49.972460    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:00:49.983379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:49.972524    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:00:50.035381 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:50.034478    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:00:50.035381 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:50.034548    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:00:52.881321 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:52.881288    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:58.927627 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.927550    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"]
Nov 02 15:00:58.928022 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.927964    4492 cpu_manager.go:414] "containerMap: removing container" podUID="83732677-acea-47fa-9b17-486c2f33efc9" containerName="revoke-keys"
Nov 02 15:00:58.928022 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.927976    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83732677-acea-47fa-9b17-486c2f33efc9" containerName="revoke-keys"
Nov 02 15:00:58.928108 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.928058    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="83732677-acea-47fa-9b17-486c2f33efc9" containerName="revoke-keys"
Nov 02 15:00:58.931294 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.931269    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:58.951839 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.951809    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"]
Nov 02 15:00:58.959224 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.959192    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:00:58.959224 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.959220    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:00:58.959447 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.959206    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:00:58.959560 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.959541    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:00:58.984904 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.984865    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwsfg\" (UniqueName: \"kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:58.985107 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.984917    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:58.985107 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:58.985063    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.086437 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.086400    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwsfg\" (UniqueName: \"kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.086437 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.086438    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.086706 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.086476    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.087110 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.087088    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.089462 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.089437    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.108972 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.108938    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwsfg\" (UniqueName: \"kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg\") pod \"maas-api-revoke-sub-c1e271a9-rx4qc\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.246503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.246412    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:00:59.590212 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.590083    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"]
Nov 02 15:00:59.593506 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:00:59.593475    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1541d59_07f6_46f3_b7f0_7128d43e27b6.slice/crio-a541d37ad7469ea7045bc3399c19458a08bf0aa885ebe74ae1a3e29e281cb6e0 WatchSource:0}: Error finding container a541d37ad7469ea7045bc3399c19458a08bf0aa885ebe74ae1a3e29e281cb6e0: Status 404 returned error can't find the container with id a541d37ad7469ea7045bc3399c19458a08bf0aa885ebe74ae1a3e29e281cb6e0
Nov 02 15:00:59.699422 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:00:59.699391    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc" event={"ID":"a1541d59-07f6-46f3-b7f0-7128d43e27b6","Type":"ContainerStarted","Data":"a541d37ad7469ea7045bc3399c19458a08bf0aa885ebe74ae1a3e29e281cb6e0"}
Nov 02 15:01:00.707069 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:00.707033    4492 generic.go:356] "Generic (PLEG): container finished" podID="a1541d59-07f6-46f3-b7f0-7128d43e27b6" containerID="b461b7424364f60aa5e2bbe882faa2eecbdca3c14b643febb48504defb36ca3c" exitCode=0
Nov 02 15:01:00.707543 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:00.707121    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc" event={"ID":"a1541d59-07f6-46f3-b7f0-7128d43e27b6","Type":"ContainerDied","Data":"b461b7424364f60aa5e2bbe882faa2eecbdca3c14b643febb48504defb36ca3c"}
Nov 02 15:01:01.851926 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.851899    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:01:01.916067 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.916025    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca\") pod \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") "
Nov 02 15:01:01.916288 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.916128    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca\") pod \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") "
Nov 02 15:01:01.916288 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.916180    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg\" (UniqueName: \"kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg\") pod \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\" (UID: \"a1541d59-07f6-46f3-b7f0-7128d43e27b6\") "
Nov 02 15:01:01.916545 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.916513    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca" pod "a1541d59-07f6-46f3-b7f0-7128d43e27b6" (UID: "a1541d59-07f6-46f3-b7f0-7128d43e27b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:01:01.916662 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.916638    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:01.918426 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.918393    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca" pod "a1541d59-07f6-46f3-b7f0-7128d43e27b6" (UID: "a1541d59-07f6-46f3-b7f0-7128d43e27b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:01:01.918525 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:01.918471    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg" pod "a1541d59-07f6-46f3-b7f0-7128d43e27b6" (UID: "a1541d59-07f6-46f3-b7f0-7128d43e27b6"). InnerVolumeSpecName "kube-api-access-nwsfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:02.017275 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:02.017187    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1541d59-07f6-46f3-b7f0-7128d43e27b6-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:02.017275 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:02.017220    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwsfg\" (UniqueName: \"kubernetes.io/projected/a1541d59-07f6-46f3-b7f0-7128d43e27b6-kube-api-access-nwsfg\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:02.719557 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:02.719528    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"
Nov 02 15:01:02.719744 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:02.719567    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc" event={"ID":"a1541d59-07f6-46f3-b7f0-7128d43e27b6","Type":"ContainerDied","Data":"a541d37ad7469ea7045bc3399c19458a08bf0aa885ebe74ae1a3e29e281cb6e0"}
Nov 02 15:01:02.719744 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:02.719620    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a541d37ad7469ea7045bc3399c19458a08bf0aa885ebe74ae1a3e29e281cb6e0"
Nov 02 15:01:10.398268 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.398215    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"]
Nov 02 15:01:10.398875 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.398851    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a1541d59-07f6-46f3-b7f0-7128d43e27b6" containerName="revoke-keys"
Nov 02 15:01:10.398961 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.398880    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1541d59-07f6-46f3-b7f0-7128d43e27b6" containerName="revoke-keys"
Nov 02 15:01:10.399046 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.399033    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1541d59-07f6-46f3-b7f0-7128d43e27b6" containerName="revoke-keys"
Nov 02 15:01:10.402527 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.402498    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.422754 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.422718    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"]
Nov 02 15:01:10.426986 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.426954    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:01:10.427143 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.427086    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:01:10.427221 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.427146    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:01:10.427221 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.427197    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:01:10.507001 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.506959    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.507202 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.507089    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8tq\" (UniqueName: \"kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.507279 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.507253    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.608235 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.608195    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.608481 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.608281    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.608481 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.608327    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8tq\" (UniqueName: \"kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.609157 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.609076    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.611298 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.611275    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.630615 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.630584    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8tq\" (UniqueName: \"kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq\") pod \"maas-api-revoke-sub-ec843018-29tkg\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:10.723074 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:10.723028    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:11.043316 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:11.043241    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"]
Nov 02 15:01:11.047178 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:01:11.047147    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f3c0a3_90ab_41b1_b95a_8cb5af424f99.slice/crio-a770859f383f8541821e344163e0296f166f1c32ecd04157a6442ab47d22de00 WatchSource:0}: Error finding container a770859f383f8541821e344163e0296f166f1c32ecd04157a6442ab47d22de00: Status 404 returned error can't find the container with id a770859f383f8541821e344163e0296f166f1c32ecd04157a6442ab47d22de00
Nov 02 15:01:11.775248 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:11.775209    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg" event={"ID":"72f3c0a3-90ab-41b1-b95a-8cb5af424f99","Type":"ContainerStarted","Data":"ebe615d855732ac28587cb57da3575a7bc613e0593583f5d80ce3de28ef0779a"}
Nov 02 15:01:11.775806 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:11.775265    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg" event={"ID":"72f3c0a3-90ab-41b1-b95a-8cb5af424f99","Type":"ContainerStarted","Data":"a770859f383f8541821e344163e0296f166f1c32ecd04157a6442ab47d22de00"}
Nov 02 15:01:11.812472 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:11.812413    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg" podStartSLOduration=1.812399062 podStartE2EDuration="1.812399062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:11.811106715 +0000 UTC m=+3836.354822311" watchObservedRunningTime="2026-11-02 15:01:11.812399062 +0000 UTC m=+3836.356114657"
Nov 02 15:01:12.781962 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:12.781925    4492 generic.go:356] "Generic (PLEG): container finished" podID="72f3c0a3-90ab-41b1-b95a-8cb5af424f99" containerID="ebe615d855732ac28587cb57da3575a7bc613e0593583f5d80ce3de28ef0779a" exitCode=0
Nov 02 15:01:12.782398 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:12.782015    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg" event={"ID":"72f3c0a3-90ab-41b1-b95a-8cb5af424f99","Type":"ContainerDied","Data":"ebe615d855732ac28587cb57da3575a7bc613e0593583f5d80ce3de28ef0779a"}
Nov 02 15:01:13.951002 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:13.950969    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:14.045157 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.045058    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca\") pod \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") "
Nov 02 15:01:14.045157 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.045109    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq\" (UniqueName: \"kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq\") pod \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") "
Nov 02 15:01:14.045446 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.045236    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca\") pod \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\" (UID: \"72f3c0a3-90ab-41b1-b95a-8cb5af424f99\") "
Nov 02 15:01:14.045686 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.045655    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca" pod "72f3c0a3-90ab-41b1-b95a-8cb5af424f99" (UID: "72f3c0a3-90ab-41b1-b95a-8cb5af424f99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:01:14.047513 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.047484    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca" pod "72f3c0a3-90ab-41b1-b95a-8cb5af424f99" (UID: "72f3c0a3-90ab-41b1-b95a-8cb5af424f99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:01:14.047621 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.047566    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq" pod "72f3c0a3-90ab-41b1-b95a-8cb5af424f99" (UID: "72f3c0a3-90ab-41b1-b95a-8cb5af424f99"). InnerVolumeSpecName "kube-api-access-tp8tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:14.146780 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.146744    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:14.146780 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.146781    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp8tq\" (UniqueName: \"kubernetes.io/projected/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-kube-api-access-tp8tq\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:14.146780 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.146798    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f3c0a3-90ab-41b1-b95a-8cb5af424f99-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:14.795278 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.795238    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg" event={"ID":"72f3c0a3-90ab-41b1-b95a-8cb5af424f99","Type":"ContainerDied","Data":"a770859f383f8541821e344163e0296f166f1c32ecd04157a6442ab47d22de00"}
Nov 02 15:01:14.795278 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.795265    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"
Nov 02 15:01:14.795542 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:14.795299    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a770859f383f8541821e344163e0296f166f1c32ecd04157a6442ab47d22de00"
Nov 02 15:01:15.649666 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:15.649623    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:16.516490 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:01:16.516446    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:01:17.597547 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:17.597503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:17.934977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:17.934947    4492 scope.go:122] "RemoveContainer" containerID="232ebd3356aaa8783dd6a489891353b5780466d0193a13050dbc11fff1c92c33"
Nov 02 15:01:32.528295 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:32.528261    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:38.628083 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.628049    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"]
Nov 02 15:01:38.628521 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.628481    4492 cpu_manager.go:414] "containerMap: removing container" podUID="72f3c0a3-90ab-41b1-b95a-8cb5af424f99" containerName="revoke-keys"
Nov 02 15:01:38.628521 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.628494    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f3c0a3-90ab-41b1-b95a-8cb5af424f99" containerName="revoke-keys"
Nov 02 15:01:38.628616 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.628572    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f3c0a3-90ab-41b1-b95a-8cb5af424f99" containerName="revoke-keys"
Nov 02 15:01:38.631908 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.631885    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.650575 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.650547    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"]
Nov 02 15:01:38.656258 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.656229    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:01:38.656464 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.656376    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:01:38.656464 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.656380    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:01:38.656464 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.656410    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:01:38.750588 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.750553    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.750771 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.750667    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.750771 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.750753    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947nl\" (UniqueName: \"kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.851425 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.851381    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.851597 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.851459    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.851597 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.851528    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947nl\" (UniqueName: \"kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.852025 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.852000    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.853925 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.853906    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.873334 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.873308    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947nl\" (UniqueName: \"kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl\") pod \"maas-api-revoke-sub-046e1343-zksbt\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:38.944795 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:38.944767    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:39.242480 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:39.242397    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"]
Nov 02 15:01:39.245986 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:01:39.245957    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70d513b_a32b_4154_9d13_fcc961495719.slice/crio-0c5aa6902765906f5c266c8de8d078fa3009bef2462f4485c7d9599aff279230 WatchSource:0}: Error finding container 0c5aa6902765906f5c266c8de8d078fa3009bef2462f4485c7d9599aff279230: Status 404 returned error can't find the container with id 0c5aa6902765906f5c266c8de8d078fa3009bef2462f4485c7d9599aff279230
Nov 02 15:01:39.982843 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:39.982806    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt" event={"ID":"a70d513b-a32b-4154-9d13-fcc961495719","Type":"ContainerStarted","Data":"87ae35e49f317e574661e9f73f8f894dfa90bf363226b851f696de30984f40f6"}
Nov 02 15:01:39.983315 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:39.982899    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt" event={"ID":"a70d513b-a32b-4154-9d13-fcc961495719","Type":"ContainerStarted","Data":"0c5aa6902765906f5c266c8de8d078fa3009bef2462f4485c7d9599aff279230"}
Nov 02 15:01:40.021181 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:40.021129    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt" podStartSLOduration=2.021116422 podStartE2EDuration="2.021116422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:40.019624818 +0000 UTC m=+3864.563340412" watchObservedRunningTime="2026-11-02 15:01:40.021116422 +0000 UTC m=+3864.564832015"
Nov 02 15:01:40.990846 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:40.990814    4492 generic.go:356] "Generic (PLEG): container finished" podID="a70d513b-a32b-4154-9d13-fcc961495719" containerID="87ae35e49f317e574661e9f73f8f894dfa90bf363226b851f696de30984f40f6" exitCode=0
Nov 02 15:01:40.991248 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:40.990900    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt" event={"ID":"a70d513b-a32b-4154-9d13-fcc961495719","Type":"ContainerDied","Data":"87ae35e49f317e574661e9f73f8f894dfa90bf363226b851f696de30984f40f6"}
Nov 02 15:01:41.378307 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:41.378222    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:42.130711 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.130688    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:42.284997 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.284917    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca\") pod \"a70d513b-a32b-4154-9d13-fcc961495719\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") "
Nov 02 15:01:42.285161 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.285013    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca\") pod \"a70d513b-a32b-4154-9d13-fcc961495719\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") "
Nov 02 15:01:42.285161 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.285107    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl\" (UniqueName: \"kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl\") pod \"a70d513b-a32b-4154-9d13-fcc961495719\" (UID: \"a70d513b-a32b-4154-9d13-fcc961495719\") "
Nov 02 15:01:42.285409 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.285385    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca" pod "a70d513b-a32b-4154-9d13-fcc961495719" (UID: "a70d513b-a32b-4154-9d13-fcc961495719"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:01:42.287242 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.287219    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca" pod "a70d513b-a32b-4154-9d13-fcc961495719" (UID: "a70d513b-a32b-4154-9d13-fcc961495719"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:01:42.287410 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.287393    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl" pod "a70d513b-a32b-4154-9d13-fcc961495719" (UID: "a70d513b-a32b-4154-9d13-fcc961495719"). InnerVolumeSpecName "kube-api-access-947nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:42.386156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.386115    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947nl\" (UniqueName: \"kubernetes.io/projected/a70d513b-a32b-4154-9d13-fcc961495719-kube-api-access-947nl\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:42.386156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.386148    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:42.386156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:42.386164    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a70d513b-a32b-4154-9d13-fcc961495719-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:43.002779 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:43.002741    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"
Nov 02 15:01:43.002969 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:43.002743    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt" event={"ID":"a70d513b-a32b-4154-9d13-fcc961495719","Type":"ContainerDied","Data":"0c5aa6902765906f5c266c8de8d078fa3009bef2462f4485c7d9599aff279230"}
Nov 02 15:01:43.002969 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:43.002884    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5aa6902765906f5c266c8de8d078fa3009bef2462f4485c7d9599aff279230"
Nov 02 15:01:46.515664 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:01:46.515621    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:01:49.010772 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.010729    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"]
Nov 02 15:01:49.011206 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.011185    4492 cpu_manager.go:414] "containerMap: removing container" podUID="a70d513b-a32b-4154-9d13-fcc961495719" containerName="revoke-keys"
Nov 02 15:01:49.011206 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.011197    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70d513b-a32b-4154-9d13-fcc961495719" containerName="revoke-keys"
Nov 02 15:01:49.011297 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.011261    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70d513b-a32b-4154-9d13-fcc961495719" containerName="revoke-keys"
Nov 02 15:01:49.014823 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.014798    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.033609 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.033576    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"]
Nov 02 15:01:49.039300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.039110    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:01:49.039300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.039146    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:01:49.039300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.039198    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:01:49.039300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.039268    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:01:49.057701 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.057669    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6p6\" (UniqueName: \"kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.057898 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.057792    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.057898 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.057863    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.158800 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.158761    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6p6\" (UniqueName: \"kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.158981 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.158861    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.158981 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.158910    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.159586 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.159558    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.161588 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.161560    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.193973 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.193934    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6p6\" (UniqueName: \"kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6\") pod \"maas-api-revoke-sub-d3fba056-v29xf\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.330586 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.330496    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:49.629856 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:49.629757    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"]
Nov 02 15:01:49.632979 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:01:49.632951    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1a993f_acae_4917_881d_80ce69f09a0c.slice/crio-e5b3a75937a7be08e43cd4fe293a24b685c0f519ca31eaa7f70d8c66c7e72d37 WatchSource:0}: Error finding container e5b3a75937a7be08e43cd4fe293a24b685c0f519ca31eaa7f70d8c66c7e72d37: Status 404 returned error can't find the container with id e5b3a75937a7be08e43cd4fe293a24b685c0f519ca31eaa7f70d8c66c7e72d37
Nov 02 15:01:50.046839 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:50.046805    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf" event={"ID":"bc1a993f-acae-4917-881d-80ce69f09a0c","Type":"ContainerStarted","Data":"9468c8abe7fcd29ab6a3b7bcd937d6d9ee6486750d525c8895028ac9d9cd4517"}
Nov 02 15:01:50.047254 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:50.046859    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf" event={"ID":"bc1a993f-acae-4917-881d-80ce69f09a0c","Type":"ContainerStarted","Data":"e5b3a75937a7be08e43cd4fe293a24b685c0f519ca31eaa7f70d8c66c7e72d37"}
Nov 02 15:01:50.086758 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:50.086670    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf" podStartSLOduration=2.086651319 podStartE2EDuration="2.086651319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:50.086545608 +0000 UTC m=+3874.630261205" watchObservedRunningTime="2026-11-02 15:01:50.086651319 +0000 UTC m=+3874.630366915"
Nov 02 15:01:51.057732 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:51.057691    4492 generic.go:356] "Generic (PLEG): container finished" podID="bc1a993f-acae-4917-881d-80ce69f09a0c" containerID="9468c8abe7fcd29ab6a3b7bcd937d6d9ee6486750d525c8895028ac9d9cd4517" exitCode=0
Nov 02 15:01:51.058284 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:51.057759    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf" event={"ID":"bc1a993f-acae-4917-881d-80ce69f09a0c","Type":"ContainerDied","Data":"9468c8abe7fcd29ab6a3b7bcd937d6d9ee6486750d525c8895028ac9d9cd4517"}
Nov 02 15:01:52.863379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:52.849699    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:01:52.993380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:52.992945    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6\" (UniqueName: \"kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6\") pod \"bc1a993f-acae-4917-881d-80ce69f09a0c\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") "
Nov 02 15:01:52.993380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:52.993005    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca\") pod \"bc1a993f-acae-4917-881d-80ce69f09a0c\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") "
Nov 02 15:01:52.993380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:52.993068    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca\") pod \"bc1a993f-acae-4917-881d-80ce69f09a0c\" (UID: \"bc1a993f-acae-4917-881d-80ce69f09a0c\") "
Nov 02 15:01:53.008381 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:52.996565    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca" pod "bc1a993f-acae-4917-881d-80ce69f09a0c" (UID: "bc1a993f-acae-4917-881d-80ce69f09a0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:01:53.020067 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.020029    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6" pod "bc1a993f-acae-4917-881d-80ce69f09a0c" (UID: "bc1a993f-acae-4917-881d-80ce69f09a0c"). InnerVolumeSpecName "kube-api-access-sj6p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:53.022903 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.022870    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca" pod "bc1a993f-acae-4917-881d-80ce69f09a0c" (UID: "bc1a993f-acae-4917-881d-80ce69f09a0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:01:53.048380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.043952    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:53.159397 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.136162    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:53.159397 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.136200    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6p6\" (UniqueName: \"kubernetes.io/projected/bc1a993f-acae-4917-881d-80ce69f09a0c-kube-api-access-sj6p6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:53.159397 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.136216    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1a993f-acae-4917-881d-80ce69f09a0c-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:01:53.228325 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.228289    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf" event={"ID":"bc1a993f-acae-4917-881d-80ce69f09a0c","Type":"ContainerDied","Data":"e5b3a75937a7be08e43cd4fe293a24b685c0f519ca31eaa7f70d8c66c7e72d37"}
Nov 02 15:01:53.228651 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.228628    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b3a75937a7be08e43cd4fe293a24b685c0f519ca31eaa7f70d8c66c7e72d37"
Nov 02 15:01:53.228857 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:01:53.228843    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"
Nov 02 15:02:10.759387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.759337    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"]
Nov 02 15:02:10.760703 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.760670    4492 cpu_manager.go:414] "containerMap: removing container" podUID="bc1a993f-acae-4917-881d-80ce69f09a0c" containerName="revoke-keys"
Nov 02 15:02:10.760703 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.760701    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1a993f-acae-4917-881d-80ce69f09a0c" containerName="revoke-keys"
Nov 02 15:02:10.761300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.760779    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1a993f-acae-4917-881d-80ce69f09a0c" containerName="revoke-keys"
Nov 02 15:02:10.763841 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.763816    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.779869 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.779846    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:02:10.786134 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.786101    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:02:10.786270 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.786133    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:02:10.786270 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.786106    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:02:10.789882 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.789857    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"]
Nov 02 15:02:10.885220 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.885184    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.885447 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.885225    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.885447 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.885248    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5mn\" (UniqueName: \"kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.986727 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.986688    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.986934 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.986739    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.986934 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.986769    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5mn\" (UniqueName: \"kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.987469 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.987448    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:10.989290 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:10.989271    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:11.008412 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:11.008381    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5mn\" (UniqueName: \"kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn\") pod \"maas-api-revoke-sub-ae594979-cp8q4\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:11.084529 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:11.084448    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:11.389239 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:11.389154    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"]
Nov 02 15:02:11.392473 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:02:11.392442    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cfb406_4ece_42a7_9d05_26b862f9c3ac.slice/crio-e64a40bfb2612b86c2091dd0e8d63661b19d391ec3b7d4153c07969839ffb67c WatchSource:0}: Error finding container e64a40bfb2612b86c2091dd0e8d63661b19d391ec3b7d4153c07969839ffb67c: Status 404 returned error can't find the container with id e64a40bfb2612b86c2091dd0e8d63661b19d391ec3b7d4153c07969839ffb67c
Nov 02 15:02:12.366164 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:12.366126    4492 generic.go:356] "Generic (PLEG): container finished" podID="37cfb406-4ece-42a7-9d05-26b862f9c3ac" containerID="df54805da4aecc12a5b5044596d8f23461a3b3d0829bb29e32149a1970af09d9" exitCode=0
Nov 02 15:02:12.366625 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:12.366212    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4" event={"ID":"37cfb406-4ece-42a7-9d05-26b862f9c3ac","Type":"ContainerDied","Data":"df54805da4aecc12a5b5044596d8f23461a3b3d0829bb29e32149a1970af09d9"}
Nov 02 15:02:12.366625 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:12.366265    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4" event={"ID":"37cfb406-4ece-42a7-9d05-26b862f9c3ac","Type":"ContainerStarted","Data":"e64a40bfb2612b86c2091dd0e8d63661b19d391ec3b7d4153c07969839ffb67c"}
Nov 02 15:02:13.510510 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.510482    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:13.615568 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.615528    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca\") pod \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") "
Nov 02 15:02:13.615779 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.615620    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn\" (UniqueName: \"kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn\") pod \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") "
Nov 02 15:02:13.615779 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.615733    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca\") pod \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\" (UID: \"37cfb406-4ece-42a7-9d05-26b862f9c3ac\") "
Nov 02 15:02:13.616082 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.616059    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca" pod "37cfb406-4ece-42a7-9d05-26b862f9c3ac" (UID: "37cfb406-4ece-42a7-9d05-26b862f9c3ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:02:13.617851 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.617810    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca" pod "37cfb406-4ece-42a7-9d05-26b862f9c3ac" (UID: "37cfb406-4ece-42a7-9d05-26b862f9c3ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:13.617953 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.617872    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn" pod "37cfb406-4ece-42a7-9d05-26b862f9c3ac" (UID: "37cfb406-4ece-42a7-9d05-26b862f9c3ac"). InnerVolumeSpecName "kube-api-access-fs5mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:02:13.716807 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.716758    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs5mn\" (UniqueName: \"kubernetes.io/projected/37cfb406-4ece-42a7-9d05-26b862f9c3ac-kube-api-access-fs5mn\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:13.716807 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.716801    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:13.716807 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:13.716816    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cfb406-4ece-42a7-9d05-26b862f9c3ac-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:14.378429 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:14.378388    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4" event={"ID":"37cfb406-4ece-42a7-9d05-26b862f9c3ac","Type":"ContainerDied","Data":"e64a40bfb2612b86c2091dd0e8d63661b19d391ec3b7d4153c07969839ffb67c"}
Nov 02 15:02:14.378647 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:14.378454    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64a40bfb2612b86c2091dd0e8d63661b19d391ec3b7d4153c07969839ffb67c"
Nov 02 15:02:14.378647 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:14.378402    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"
Nov 02 15:02:16.517337 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:02:16.517297    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:02:17.715061 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:17.715012    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:28.455426 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.455333    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"]
Nov 02 15:02:28.455818 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.455804    4492 cpu_manager.go:414] "containerMap: removing container" podUID="37cfb406-4ece-42a7-9d05-26b862f9c3ac" containerName="revoke-keys"
Nov 02 15:02:28.455873 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.455820    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cfb406-4ece-42a7-9d05-26b862f9c3ac" containerName="revoke-keys"
Nov 02 15:02:28.455911 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.455899    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cfb406-4ece-42a7-9d05-26b862f9c3ac" containerName="revoke-keys"
Nov 02 15:02:28.459256 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.459224    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.482001 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.481973    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:02:28.482208 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.481985    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:02:28.482306 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.482082    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:02:28.482306 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.482187    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:02:28.486777 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.486748    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"]
Nov 02 15:02:28.528304 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.528271    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmc8\" (UniqueName: \"kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.528503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.528486    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.528550 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.528526    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.629053 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.629012    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.629263 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.629065    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.629263 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.629092    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmc8\" (UniqueName: \"kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.629783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.629759    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.631623 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.631597    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.669969 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.669933    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmc8\" (UniqueName: \"kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8\") pod \"maas-api-revoke-sub-a4aef7d7-bmgpk\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:28.773726 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:28.773648    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:29.084689 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:02:29.084646    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966cd667_33eb_449b_bfd6_d26f334996ab.slice/crio-c6a552b0fa4e95d0d74e71e63ba38464cedbabc6cb530628b041e91f3c12a496 WatchSource:0}: Error finding container c6a552b0fa4e95d0d74e71e63ba38464cedbabc6cb530628b041e91f3c12a496: Status 404 returned error can't find the container with id c6a552b0fa4e95d0d74e71e63ba38464cedbabc6cb530628b041e91f3c12a496
Nov 02 15:02:29.089631 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:29.089609    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"]
Nov 02 15:02:29.497895 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:29.497854    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk" event={"ID":"966cd667-33eb-449b-bfd6-d26f334996ab","Type":"ContainerStarted","Data":"30ee4114a72dffa70e0c18c3ba0e38536f4e0a11a74a2102a99fc4b70c557117"}
Nov 02 15:02:29.498298 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:29.497921    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk" event={"ID":"966cd667-33eb-449b-bfd6-d26f334996ab","Type":"ContainerStarted","Data":"c6a552b0fa4e95d0d74e71e63ba38464cedbabc6cb530628b041e91f3c12a496"}
Nov 02 15:02:30.504759 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:30.504728    4492 generic.go:356] "Generic (PLEG): container finished" podID="966cd667-33eb-449b-bfd6-d26f334996ab" containerID="30ee4114a72dffa70e0c18c3ba0e38536f4e0a11a74a2102a99fc4b70c557117" exitCode=0
Nov 02 15:02:30.505157 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:30.504816    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk" event={"ID":"966cd667-33eb-449b-bfd6-d26f334996ab","Type":"ContainerDied","Data":"30ee4114a72dffa70e0c18c3ba0e38536f4e0a11a74a2102a99fc4b70c557117"}
Nov 02 15:02:31.651712 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.651687    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:31.762721 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.762627    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca\") pod \"966cd667-33eb-449b-bfd6-d26f334996ab\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") "
Nov 02 15:02:31.762897 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.762767    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca\") pod \"966cd667-33eb-449b-bfd6-d26f334996ab\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") "
Nov 02 15:02:31.762897 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.762891    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8\" (UniqueName: \"kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8\") pod \"966cd667-33eb-449b-bfd6-d26f334996ab\" (UID: \"966cd667-33eb-449b-bfd6-d26f334996ab\") "
Nov 02 15:02:31.763117 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.763093    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca" pod "966cd667-33eb-449b-bfd6-d26f334996ab" (UID: "966cd667-33eb-449b-bfd6-d26f334996ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:02:31.763260 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.763238    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:31.764929 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.764907    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca" pod "966cd667-33eb-449b-bfd6-d26f334996ab" (UID: "966cd667-33eb-449b-bfd6-d26f334996ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:31.765050 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.765031    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8" pod "966cd667-33eb-449b-bfd6-d26f334996ab" (UID: "966cd667-33eb-449b-bfd6-d26f334996ab"). InnerVolumeSpecName "kube-api-access-phmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:02:31.864326 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.864290    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmc8\" (UniqueName: \"kubernetes.io/projected/966cd667-33eb-449b-bfd6-d26f334996ab-kube-api-access-phmc8\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:31.864326 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:31.864319    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966cd667-33eb-449b-bfd6-d26f334996ab-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:32.517658 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:32.517628    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"
Nov 02 15:02:32.517858 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:32.517659    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk" event={"ID":"966cd667-33eb-449b-bfd6-d26f334996ab","Type":"ContainerDied","Data":"c6a552b0fa4e95d0d74e71e63ba38464cedbabc6cb530628b041e91f3c12a496"}
Nov 02 15:02:32.517858 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:32.517708    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a552b0fa4e95d0d74e71e63ba38464cedbabc6cb530628b041e91f3c12a496"
Nov 02 15:02:32.683703 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:32.683661    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:45.656380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.654908    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"]
Nov 02 15:02:45.656380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.655593    4492 cpu_manager.go:414] "containerMap: removing container" podUID="966cd667-33eb-449b-bfd6-d26f334996ab" containerName="revoke-keys"
Nov 02 15:02:45.656380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.655616    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966cd667-33eb-449b-bfd6-d26f334996ab" containerName="revoke-keys"
Nov 02 15:02:45.656380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.655738    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="966cd667-33eb-449b-bfd6-d26f334996ab" containerName="revoke-keys"
Nov 02 15:02:45.659769 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.659743    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.700021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.698824    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:02:45.700021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.698912    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:02:45.700021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.698999    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:02:45.700021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.699045    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:02:45.739714 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.739203    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"]
Nov 02 15:02:45.845383 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.843584    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.845383 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.843700    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlr8\" (UniqueName: \"kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.845383 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.843811    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.948379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.946946    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.948379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.947074    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlr8\" (UniqueName: \"kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.948379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.947188    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.948379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.948057    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:45.986380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:45.984610    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:46.018391 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:46.017494    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlr8\" (UniqueName: \"kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8\") pod \"maas-api-revoke-sub-79805ca9-l7ckg\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:46.050536 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:46.047650    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:47.180380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:47.164236    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"]
Nov 02 15:02:47.298331 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:02:47.224421    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6070219e_d570_486c_92f0_443d5f11bb36.slice/crio-433b0f809c1b765065bbf08745f5253ffa1c817c4b35e36f4d518e12317b625e WatchSource:0}: Error finding container 433b0f809c1b765065bbf08745f5253ffa1c817c4b35e36f4d518e12317b625e: Status 404 returned error can't find the container with id 433b0f809c1b765065bbf08745f5253ffa1c817c4b35e36f4d518e12317b625e
Nov 02 15:02:47.530449 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:02:47.527319    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:02:47.821621 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:47.821522    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg" event={"ID":"6070219e-d570-486c-92f0-443d5f11bb36","Type":"ContainerStarted","Data":"433b0f809c1b765065bbf08745f5253ffa1c817c4b35e36f4d518e12317b625e"}
Nov 02 15:02:48.829317 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:48.829228    4492 generic.go:356] "Generic (PLEG): container finished" podID="6070219e-d570-486c-92f0-443d5f11bb36" containerID="0c1e848b8f01fea8bf4b542eb3561f7c45ba1a1244762c22bc22a27a20df4e4c" exitCode=0
Nov 02 15:02:48.829317 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:48.829302    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg" event={"ID":"6070219e-d570-486c-92f0-443d5f11bb36","Type":"ContainerDied","Data":"0c1e848b8f01fea8bf4b542eb3561f7c45ba1a1244762c22bc22a27a20df4e4c"}
Nov 02 15:02:49.976921 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:49.976897    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:50.047418 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.046602    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca\") pod \"6070219e-d570-486c-92f0-443d5f11bb36\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") "
Nov 02 15:02:50.047777 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.047754    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8\" (UniqueName: \"kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8\") pod \"6070219e-d570-486c-92f0-443d5f11bb36\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") "
Nov 02 15:02:50.047960 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.047943    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca\") pod \"6070219e-d570-486c-92f0-443d5f11bb36\" (UID: \"6070219e-d570-486c-92f0-443d5f11bb36\") "
Nov 02 15:02:50.049001 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.048976    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca" pod "6070219e-d570-486c-92f0-443d5f11bb36" (UID: "6070219e-d570-486c-92f0-443d5f11bb36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:02:50.051651 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.051593    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8" pod "6070219e-d570-486c-92f0-443d5f11bb36" (UID: "6070219e-d570-486c-92f0-443d5f11bb36"). InnerVolumeSpecName "kube-api-access-fxlr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:02:50.051978 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.051938    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca" pod "6070219e-d570-486c-92f0-443d5f11bb36" (UID: "6070219e-d570-486c-92f0-443d5f11bb36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:02:50.149393 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.149302    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:50.149393 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.149328    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6070219e-d570-486c-92f0-443d5f11bb36-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:50.149393 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.149338    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxlr8\" (UniqueName: \"kubernetes.io/projected/6070219e-d570-486c-92f0-443d5f11bb36-kube-api-access-fxlr8\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:02:50.843491 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.843462    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"
Nov 02 15:02:50.843491 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.843479    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg" event={"ID":"6070219e-d570-486c-92f0-443d5f11bb36","Type":"ContainerDied","Data":"433b0f809c1b765065bbf08745f5253ffa1c817c4b35e36f4d518e12317b625e"}
Nov 02 15:02:50.843805 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:50.843531    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433b0f809c1b765065bbf08745f5253ffa1c817c4b35e36f4d518e12317b625e"
Nov 02 15:02:51.299978 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:51.299939    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:59.495405 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.495341    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"]
Nov 02 15:02:59.496698 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.496670    4492 cpu_manager.go:414] "containerMap: removing container" podUID="6070219e-d570-486c-92f0-443d5f11bb36" containerName="revoke-keys"
Nov 02 15:02:59.496880 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.496864    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6070219e-d570-486c-92f0-443d5f11bb36" containerName="revoke-keys"
Nov 02 15:02:59.497100 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.497084    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="6070219e-d570-486c-92f0-443d5f11bb36" containerName="revoke-keys"
Nov 02 15:02:59.522705 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.522666    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.524919 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.524889    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"]
Nov 02 15:02:59.538870 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.538837    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:02:59.539148 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.539131    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:02:59.549695 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.549662    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:02:59.565382 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.550692    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:02:59.702392 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.695785    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.702392 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.695853    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.702392 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.699562    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfbnq\" (UniqueName: \"kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.813430 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.809337    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.890481 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.890449    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.890902 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.890874    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfbnq\" (UniqueName: \"kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.892171 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.892146    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.953171 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.953128    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfbnq\" (UniqueName: \"kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:02:59.994382 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:02:59.981795    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4pmxc\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:03:00.178643 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:00.178595    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:03:00.515440 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:00.512936    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:01.163664 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:01.163581    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"]
Nov 02 15:03:01.269690 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:03:01.269650    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda365474_339d_43ed_8928_f6f440947989.slice/crio-6b3dd98d9926b42c98deadad9f2db675e3a9d1871447ed9ad50dbf24addd33a3 WatchSource:0}: Error finding container 6b3dd98d9926b42c98deadad9f2db675e3a9d1871447ed9ad50dbf24addd33a3: Status 404 returned error can't find the container with id 6b3dd98d9926b42c98deadad9f2db675e3a9d1871447ed9ad50dbf24addd33a3
Nov 02 15:03:02.078208 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:02.078112    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc" event={"ID":"da365474-339d-43ed-8928-f6f440947989","Type":"ContainerStarted","Data":"9989fb78b9cc393f5e81a0feabfe0be87d17d7195e3b98aa3de90b4f9715527a"}
Nov 02 15:03:02.078208 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:02.078180    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc" event={"ID":"da365474-339d-43ed-8928-f6f440947989","Type":"ContainerStarted","Data":"6b3dd98d9926b42c98deadad9f2db675e3a9d1871447ed9ad50dbf24addd33a3"}
Nov 02 15:03:02.117542 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:02.117471    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc" podStartSLOduration=3.117451204 podStartE2EDuration="3.117451204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:02.115422791 +0000 UTC m=+3946.659138387" watchObservedRunningTime="2026-11-02 15:03:02.117451204 +0000 UTC m=+3946.661166795"
Nov 02 15:03:03.086474 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:03.086438    4492 generic.go:356] "Generic (PLEG): container finished" podID="da365474-339d-43ed-8928-f6f440947989" containerID="9989fb78b9cc393f5e81a0feabfe0be87d17d7195e3b98aa3de90b4f9715527a" exitCode=0
Nov 02 15:03:03.086956 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:03.086500    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc" event={"ID":"da365474-339d-43ed-8928-f6f440947989","Type":"ContainerDied","Data":"9989fb78b9cc393f5e81a0feabfe0be87d17d7195e3b98aa3de90b4f9715527a"}
Nov 02 15:03:03.134693 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:03.134658    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:04.233797 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.233773    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:03:04.327194 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.327159    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq\" (UniqueName: \"kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq\") pod \"da365474-339d-43ed-8928-f6f440947989\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") "
Nov 02 15:03:04.327433 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.327210    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca\") pod \"da365474-339d-43ed-8928-f6f440947989\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") "
Nov 02 15:03:04.327433 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.327264    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca\") pod \"da365474-339d-43ed-8928-f6f440947989\" (UID: \"da365474-339d-43ed-8928-f6f440947989\") "
Nov 02 15:03:04.327679 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.327655    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca" pod "da365474-339d-43ed-8928-f6f440947989" (UID: "da365474-339d-43ed-8928-f6f440947989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:03:04.329557 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.329530    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca" pod "da365474-339d-43ed-8928-f6f440947989" (UID: "da365474-339d-43ed-8928-f6f440947989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:04.329629 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.329558    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq" pod "da365474-339d-43ed-8928-f6f440947989" (UID: "da365474-339d-43ed-8928-f6f440947989"). InnerVolumeSpecName "kube-api-access-jfbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:04.428779 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.428747    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfbnq\" (UniqueName: \"kubernetes.io/projected/da365474-339d-43ed-8928-f6f440947989-kube-api-access-jfbnq\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:04.428940 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.428777    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da365474-339d-43ed-8928-f6f440947989-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:04.428940 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:04.428801    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da365474-339d-43ed-8928-f6f440947989-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:05.098851 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:05.098818    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"
Nov 02 15:03:05.099037 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:05.098865    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc" event={"ID":"da365474-339d-43ed-8928-f6f440947989","Type":"ContainerDied","Data":"6b3dd98d9926b42c98deadad9f2db675e3a9d1871447ed9ad50dbf24addd33a3"}
Nov 02 15:03:05.099037 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:05.098912    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3dd98d9926b42c98deadad9f2db675e3a9d1871447ed9ad50dbf24addd33a3"
Nov 02 15:03:10.225818 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.225779    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"]
Nov 02 15:03:10.226290 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.226275    4492 cpu_manager.go:414] "containerMap: removing container" podUID="da365474-339d-43ed-8928-f6f440947989" containerName="revoke-keys"
Nov 02 15:03:10.226341 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.226293    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da365474-339d-43ed-8928-f6f440947989" containerName="revoke-keys"
Nov 02 15:03:10.226402 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.226395    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="da365474-339d-43ed-8928-f6f440947989" containerName="revoke-keys"
Nov 02 15:03:10.229902 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.229879    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.246787 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.246759    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:03:10.246978 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.246887    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:03:10.249662 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.249635    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"]
Nov 02 15:03:10.256173 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.256147    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:03:10.256173 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.256162    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:03:10.289045 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.289015    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.289240 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.289067    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.289240 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.289169    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4m9\" (UniqueName: \"kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.390662 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.390621    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.390867 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.390675    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4m9\" (UniqueName: \"kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.390867 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.390823    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.391583 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.391500    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.393250 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.393228    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.413010 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.412976    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4m9\" (UniqueName: \"kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9\") pod \"maas-api-revoke-sub-3d4156f1-qrwtn\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.543309 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.543227    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:10.846548 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:10.846467    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"]
Nov 02 15:03:10.850412 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:03:10.850383    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dafbcae_1d77_4800_b84b_a1d1ab0f372c.slice/crio-ec2ba9efde7de9412f8b498d46b05355d475f84b648100971fb21b7c8f79bfc0 WatchSource:0}: Error finding container ec2ba9efde7de9412f8b498d46b05355d475f84b648100971fb21b7c8f79bfc0: Status 404 returned error can't find the container with id ec2ba9efde7de9412f8b498d46b05355d475f84b648100971fb21b7c8f79bfc0
Nov 02 15:03:11.138465 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:11.138432    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn" event={"ID":"5dafbcae-1d77-4800-b84b-a1d1ab0f372c","Type":"ContainerStarted","Data":"ec2ba9efde7de9412f8b498d46b05355d475f84b648100971fb21b7c8f79bfc0"}
Nov 02 15:03:12.146026 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:12.145986    4492 generic.go:356] "Generic (PLEG): container finished" podID="5dafbcae-1d77-4800-b84b-a1d1ab0f372c" containerID="d370593d6cd669fc071bf6052de9b252bd0a574a7dd1c03d40aa628519a58791" exitCode=0
Nov 02 15:03:12.146514 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:12.146075    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn" event={"ID":"5dafbcae-1d77-4800-b84b-a1d1ab0f372c","Type":"ContainerDied","Data":"d370593d6cd669fc071bf6052de9b252bd0a574a7dd1c03d40aa628519a58791"}
Nov 02 15:03:13.289874 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.289845    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:13.323483 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.323445    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca\") pod \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") "
Nov 02 15:03:13.323634 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.323596    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca\") pod \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") "
Nov 02 15:03:13.323705 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.323674    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9\" (UniqueName: \"kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9\") pod \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\" (UID: \"5dafbcae-1d77-4800-b84b-a1d1ab0f372c\") "
Nov 02 15:03:13.324017 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.323984    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca" pod "5dafbcae-1d77-4800-b84b-a1d1ab0f372c" (UID: "5dafbcae-1d77-4800-b84b-a1d1ab0f372c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:03:13.325753 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.325731    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9" pod "5dafbcae-1d77-4800-b84b-a1d1ab0f372c" (UID: "5dafbcae-1d77-4800-b84b-a1d1ab0f372c"). InnerVolumeSpecName "kube-api-access-rp4m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:13.325884 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.325784    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca" pod "5dafbcae-1d77-4800-b84b-a1d1ab0f372c" (UID: "5dafbcae-1d77-4800-b84b-a1d1ab0f372c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:13.424548 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.424510    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:13.424548 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.424549    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:13.424548 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:13.424559    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4m9\" (UniqueName: \"kubernetes.io/projected/5dafbcae-1d77-4800-b84b-a1d1ab0f372c-kube-api-access-rp4m9\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:14.158765 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:14.158733    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"
Nov 02 15:03:14.158950 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:14.158729    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn" event={"ID":"5dafbcae-1d77-4800-b84b-a1d1ab0f372c","Type":"ContainerDied","Data":"ec2ba9efde7de9412f8b498d46b05355d475f84b648100971fb21b7c8f79bfc0"}
Nov 02 15:03:14.158950 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:14.158859    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2ba9efde7de9412f8b498d46b05355d475f84b648100971fb21b7c8f79bfc0"
Nov 02 15:03:15.376629 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:15.376595    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:15.488014 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:15.487981    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:16.517498 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:03:16.517461    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:03:27.852380 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:27.852066    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:28.636064 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.636028    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"]
Nov 02 15:03:28.645647 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.645619    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qrwtn"]
Nov 02 15:03:28.688966 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.688934    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"]
Nov 02 15:03:28.689582 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.689564    4492 cpu_manager.go:414] "containerMap: removing container" podUID="5dafbcae-1d77-4800-b84b-a1d1ab0f372c" containerName="revoke-keys"
Nov 02 15:03:28.689705 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.689584    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dafbcae-1d77-4800-b84b-a1d1ab0f372c" containerName="revoke-keys"
Nov 02 15:03:28.689750 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.689724    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dafbcae-1d77-4800-b84b-a1d1ab0f372c" containerName="revoke-keys"
Nov 02 15:03:28.693488 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.693467    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.708640 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.708614    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:03:28.708783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.708640    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:03:28.708783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.708662    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:03:28.708783 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.708701    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:03:28.712257 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.712237    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"]
Nov 02 15:03:28.762018 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.761985    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.762018 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.762031    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nf29\" (UniqueName: \"kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.762264 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.762168    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.863723 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.863686    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.863723 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.863728    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nf29\" (UniqueName: \"kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.864314 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.863775    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.864522 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.864501    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.866289 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.866262    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:28.885657 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:28.885631    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nf29\" (UniqueName: \"kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29\") pod \"maas-api-revoke-sub-3d4156f1-f57x8\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:29.008655 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:29.008627    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:29.321824 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:29.321739    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"]
Nov 02 15:03:29.325213 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:03:29.325182    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb308670a_b98f_4b42_acb1_d4f002b39edb.slice/crio-41a64429fd21373f2bc9a0afc21703f22bc4c4353b45029e0d6c8ebe9ebf737d WatchSource:0}: Error finding container 41a64429fd21373f2bc9a0afc21703f22bc4c4353b45029e0d6c8ebe9ebf737d: Status 404 returned error can't find the container with id 41a64429fd21373f2bc9a0afc21703f22bc4c4353b45029e0d6c8ebe9ebf737d
Nov 02 15:03:29.580929 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:29.580852    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dafbcae-1d77-4800-b84b-a1d1ab0f372c" path="/var/lib/kubelet/pods/5dafbcae-1d77-4800-b84b-a1d1ab0f372c/volumes"
Nov 02 15:03:30.304055 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:30.304024    4492 generic.go:356] "Generic (PLEG): container finished" podID="b308670a-b98f-4b42-acb1-d4f002b39edb" containerID="bd5e96460489969828602fdfba5bd33d95afdedd42bbffea3248906bebc3755d" exitCode=0
Nov 02 15:03:30.304479 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:30.304123    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8" event={"ID":"b308670a-b98f-4b42-acb1-d4f002b39edb","Type":"ContainerDied","Data":"bd5e96460489969828602fdfba5bd33d95afdedd42bbffea3248906bebc3755d"}
Nov 02 15:03:30.304479 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:30.304189    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8" event={"ID":"b308670a-b98f-4b42-acb1-d4f002b39edb","Type":"ContainerStarted","Data":"41a64429fd21373f2bc9a0afc21703f22bc4c4353b45029e0d6c8ebe9ebf737d"}
Nov 02 15:03:31.453368 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.453322    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:31.592605 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.592526    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca\") pod \"b308670a-b98f-4b42-acb1-d4f002b39edb\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") "
Nov 02 15:03:31.592605 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.592596    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29\" (UniqueName: \"kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29\") pod \"b308670a-b98f-4b42-acb1-d4f002b39edb\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") "
Nov 02 15:03:31.592813 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.592698    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca\") pod \"b308670a-b98f-4b42-acb1-d4f002b39edb\" (UID: \"b308670a-b98f-4b42-acb1-d4f002b39edb\") "
Nov 02 15:03:31.593054 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.593030    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca" pod "b308670a-b98f-4b42-acb1-d4f002b39edb" (UID: "b308670a-b98f-4b42-acb1-d4f002b39edb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:03:31.593141 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.593128    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:31.594858 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.594836    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca" pod "b308670a-b98f-4b42-acb1-d4f002b39edb" (UID: "b308670a-b98f-4b42-acb1-d4f002b39edb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:31.594973 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.594892    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29" pod "b308670a-b98f-4b42-acb1-d4f002b39edb" (UID: "b308670a-b98f-4b42-acb1-d4f002b39edb"). InnerVolumeSpecName "kube-api-access-9nf29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:31.693964 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.693932    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nf29\" (UniqueName: \"kubernetes.io/projected/b308670a-b98f-4b42-acb1-d4f002b39edb-kube-api-access-9nf29\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:31.693964 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:31.693961    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b308670a-b98f-4b42-acb1-d4f002b39edb-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:03:32.318644 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:32.318615    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"
Nov 02 15:03:32.318644 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:32.318637    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8" event={"ID":"b308670a-b98f-4b42-acb1-d4f002b39edb","Type":"ContainerDied","Data":"41a64429fd21373f2bc9a0afc21703f22bc4c4353b45029e0d6c8ebe9ebf737d"}
Nov 02 15:03:32.318929 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:32.318680    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a64429fd21373f2bc9a0afc21703f22bc4c4353b45029e0d6c8ebe9ebf737d"
Nov 02 15:03:32.795537 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:03:32.795492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:46.516632 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:03:46.516590    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:04:03.408408 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:03.403192    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:07.776277 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.776241    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"]
Nov 02 15:04:07.776721 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.776679    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b308670a-b98f-4b42-acb1-d4f002b39edb" containerName="revoke-keys"
Nov 02 15:04:07.776721 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.776693    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b308670a-b98f-4b42-acb1-d4f002b39edb" containerName="revoke-keys"
Nov 02 15:04:07.776810 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.776787    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="b308670a-b98f-4b42-acb1-d4f002b39edb" containerName="revoke-keys"
Nov 02 15:04:07.780233 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.780211    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.799114 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.799088    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"]
Nov 02 15:04:07.804485 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.804458    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:04:07.804687 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.804486    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:04:07.804687 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.804505    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:04:07.804687 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.804532    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:04:07.896062 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.896029    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.896280 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.896180    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbnq\" (UniqueName: \"kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.896399 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.896313    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.997522 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.997483    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.997743 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.997573    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbnq\" (UniqueName: \"kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.997743 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.997628    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:07.998229 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:07.998202    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:08.000071 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:08.000050    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:08.019257 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:08.019230    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbnq\" (UniqueName: \"kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq\") pod \"maas-api-revoke-sub-e2fc9c5b-blcrx\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:08.093298 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:08.093213    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:08.398950 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:08.398781    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"]
Nov 02 15:04:08.401807 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:04:08.401778    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c571ec_8c92_405f_8bde_2b78d48bd3e9.slice/crio-0ddcd43d21e5139411bcce4a525ab671b13f06a3d32cf3f04c0a3b6529e0ea0d WatchSource:0}: Error finding container 0ddcd43d21e5139411bcce4a525ab671b13f06a3d32cf3f04c0a3b6529e0ea0d: Status 404 returned error can't find the container with id 0ddcd43d21e5139411bcce4a525ab671b13f06a3d32cf3f04c0a3b6529e0ea0d
Nov 02 15:04:08.571830 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:08.571797    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx" event={"ID":"b5c571ec-8c92-405f-8bde-2b78d48bd3e9","Type":"ContainerStarted","Data":"0ddcd43d21e5139411bcce4a525ab671b13f06a3d32cf3f04c0a3b6529e0ea0d"}
Nov 02 15:04:09.579789 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:09.579748    4492 generic.go:356] "Generic (PLEG): container finished" podID="b5c571ec-8c92-405f-8bde-2b78d48bd3e9" containerID="b06a8a2a244d2abbf1f18658d55031fc8d6ce52fa9e514a3117c7ad8a5607358" exitCode=0
Nov 02 15:04:09.580185 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:09.579829    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx" event={"ID":"b5c571ec-8c92-405f-8bde-2b78d48bd3e9","Type":"ContainerDied","Data":"b06a8a2a244d2abbf1f18658d55031fc8d6ce52fa9e514a3117c7ad8a5607358"}
Nov 02 15:04:10.723819 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.723791    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:10.828912 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.828879    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca\") pod \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") "
Nov 02 15:04:10.829074 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.828982    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq\" (UniqueName: \"kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq\") pod \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") "
Nov 02 15:04:10.829133 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.829093    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca\") pod \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\" (UID: \"b5c571ec-8c92-405f-8bde-2b78d48bd3e9\") "
Nov 02 15:04:10.829369 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.829324    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca" pod "b5c571ec-8c92-405f-8bde-2b78d48bd3e9" (UID: "b5c571ec-8c92-405f-8bde-2b78d48bd3e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:04:10.829508 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.829483    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:10.831325 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.831296    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca" pod "b5c571ec-8c92-405f-8bde-2b78d48bd3e9" (UID: "b5c571ec-8c92-405f-8bde-2b78d48bd3e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:04:10.831572 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.831557    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq" pod "b5c571ec-8c92-405f-8bde-2b78d48bd3e9" (UID: "b5c571ec-8c92-405f-8bde-2b78d48bd3e9"). InnerVolumeSpecName "kube-api-access-ntbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:04:10.930741 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.930702    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:10.930741 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:10.930737    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntbnq\" (UniqueName: \"kubernetes.io/projected/b5c571ec-8c92-405f-8bde-2b78d48bd3e9-kube-api-access-ntbnq\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:11.591852 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:11.591815    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx" event={"ID":"b5c571ec-8c92-405f-8bde-2b78d48bd3e9","Type":"ContainerDied","Data":"0ddcd43d21e5139411bcce4a525ab671b13f06a3d32cf3f04c0a3b6529e0ea0d"}
Nov 02 15:04:11.592021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:11.591859    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"
Nov 02 15:04:11.592021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:11.591867    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ddcd43d21e5139411bcce4a525ab671b13f06a3d32cf3f04c0a3b6529e0ea0d"
Nov 02 15:04:16.516813 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:04:16.516775    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:04:18.141342 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.141298    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"]
Nov 02 15:04:18.141901 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.141812    4492 cpu_manager.go:414] "containerMap: removing container" podUID="b5c571ec-8c92-405f-8bde-2b78d48bd3e9" containerName="revoke-keys"
Nov 02 15:04:18.141901 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.141829    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c571ec-8c92-405f-8bde-2b78d48bd3e9" containerName="revoke-keys"
Nov 02 15:04:18.142025 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.141912    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c571ec-8c92-405f-8bde-2b78d48bd3e9" containerName="revoke-keys"
Nov 02 15:04:18.145287 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.145264    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.159953 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.159924    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:04:18.174721 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.174691    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:04:18.174954 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.174726    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:04:18.174954 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.174812    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:04:18.177974 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.177950    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"]
Nov 02 15:04:18.308056 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.308006    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnbr\" (UniqueName: \"kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.308261 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.308084    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.308261 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.308182    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.409712 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.409629    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.409712 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.409681    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.409918 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.409773    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnbr\" (UniqueName: \"kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.410504 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.410482    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.412170 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.412142    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.432021 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.431984    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnbr\" (UniqueName: \"kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr\") pod \"maas-api-revoke-sub-5a29efb3-wv6sn\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.461969 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.461936    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:18.766971 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:18.766937    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"]
Nov 02 15:04:18.770260 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:04:18.770235    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95049f11_6fbc_42db_88b1_1af202feca0d.slice/crio-3c7888ee70624ee3ec65270300e394a5af39c2ef7c3ea2aa8489307c176edd05 WatchSource:0}: Error finding container 3c7888ee70624ee3ec65270300e394a5af39c2ef7c3ea2aa8489307c176edd05: Status 404 returned error can't find the container with id 3c7888ee70624ee3ec65270300e394a5af39c2ef7c3ea2aa8489307c176edd05
Nov 02 15:04:19.653231 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:19.653196    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn" event={"ID":"95049f11-6fbc-42db-88b1-1af202feca0d","Type":"ContainerStarted","Data":"6fdfa96db77ec42c3d2e8c619a3c2a34fa4765b07ebd4437f6166d9b72fb5bf9"}
Nov 02 15:04:19.653672 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:19.653265    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn" event={"ID":"95049f11-6fbc-42db-88b1-1af202feca0d","Type":"ContainerStarted","Data":"3c7888ee70624ee3ec65270300e394a5af39c2ef7c3ea2aa8489307c176edd05"}
Nov 02 15:04:20.687393 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:20.686884    4492 generic.go:356] "Generic (PLEG): container finished" podID="95049f11-6fbc-42db-88b1-1af202feca0d" containerID="6fdfa96db77ec42c3d2e8c619a3c2a34fa4765b07ebd4437f6166d9b72fb5bf9" exitCode=0
Nov 02 15:04:20.687393 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:20.687117    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn" event={"ID":"95049f11-6fbc-42db-88b1-1af202feca0d","Type":"ContainerDied","Data":"6fdfa96db77ec42c3d2e8c619a3c2a34fa4765b07ebd4437f6166d9b72fb5bf9"}
Nov 02 15:04:21.527892 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.527543    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:21.568823 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.568787    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca\") pod \"95049f11-6fbc-42db-88b1-1af202feca0d\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") "
Nov 02 15:04:21.569204 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.569181    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca\") pod \"95049f11-6fbc-42db-88b1-1af202feca0d\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") "
Nov 02 15:04:21.569603 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.569585    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr\" (UniqueName: \"kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr\") pod \"95049f11-6fbc-42db-88b1-1af202feca0d\" (UID: \"95049f11-6fbc-42db-88b1-1af202feca0d\") "
Nov 02 15:04:21.572638 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.572582    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca" pod "95049f11-6fbc-42db-88b1-1af202feca0d" (UID: "95049f11-6fbc-42db-88b1-1af202feca0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:04:21.580387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.577852    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr" pod "95049f11-6fbc-42db-88b1-1af202feca0d" (UID: "95049f11-6fbc-42db-88b1-1af202feca0d"). InnerVolumeSpecName "kube-api-access-fvnbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:04:21.678814 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.678667    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnbr\" (UniqueName: \"kubernetes.io/projected/95049f11-6fbc-42db-88b1-1af202feca0d-kube-api-access-fvnbr\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:21.678814 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.678707    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:21.899080 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.898753    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn" event={"ID":"95049f11-6fbc-42db-88b1-1af202feca0d","Type":"ContainerDied","Data":"3c7888ee70624ee3ec65270300e394a5af39c2ef7c3ea2aa8489307c176edd05"}
Nov 02 15:04:21.899080 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.898818    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7888ee70624ee3ec65270300e394a5af39c2ef7c3ea2aa8489307c176edd05"
Nov 02 15:04:21.899080 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.898889    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"
Nov 02 15:04:21.926513 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.926474    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca" pod "95049f11-6fbc-42db-88b1-1af202feca0d" (UID: "95049f11-6fbc-42db-88b1-1af202feca0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:04:21.998042 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.997474    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-4q677 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:04:21.998042 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:21.997545    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-4q677" podUID="9a1f3047-805a-4404-9b58-57ffe5f98e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:04:22.020122 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:22.020082    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95049f11-6fbc-42db-88b1-1af202feca0d-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:27.973898 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:27.973853    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:32.925636 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:32.925598    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:34.428394 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.428340    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"]
Nov 02 15:04:34.438238 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.438205    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f57x8"]
Nov 02 15:04:34.478514 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.478483    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"]
Nov 02 15:04:34.478905 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.478893    4492 cpu_manager.go:414] "containerMap: removing container" podUID="95049f11-6fbc-42db-88b1-1af202feca0d" containerName="revoke-keys"
Nov 02 15:04:34.478951 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.478906    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95049f11-6fbc-42db-88b1-1af202feca0d" containerName="revoke-keys"
Nov 02 15:04:34.478996 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.478968    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="95049f11-6fbc-42db-88b1-1af202feca0d" containerName="revoke-keys"
Nov 02 15:04:34.481953 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.481929    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.497469 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.497436    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:04:34.497469 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.497452    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:04:34.497680 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.497590    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:04:34.497680 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.497660    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:04:34.501654 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.501627    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"]
Nov 02 15:04:34.545061 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.545030    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.545262 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.545075    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.545262 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.545183    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9wg\" (UniqueName: \"kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.646714 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.646681    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.646940 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.646734    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.646940 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.646773    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9wg\" (UniqueName: \"kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.647508 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.647484    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.649327 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.649293    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.667920 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.667891    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9wg\" (UniqueName: \"kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg\") pod \"maas-api-revoke-sub-3d4156f1-cpzb5\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:34.797137 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:34.797057    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:35.099212 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:35.099183    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"]
Nov 02 15:04:35.102024 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:04:35.101996    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a0b005_68cb_408d_8375_c01427527d33.slice/crio-22eb29af15fba1f0b04d6dc67019b50151947855c834e2c0f3790e33699bbf33 WatchSource:0}: Error finding container 22eb29af15fba1f0b04d6dc67019b50151947855c834e2c0f3790e33699bbf33: Status 404 returned error can't find the container with id 22eb29af15fba1f0b04d6dc67019b50151947855c834e2c0f3790e33699bbf33
Nov 02 15:04:35.579338 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:35.579307    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b308670a-b98f-4b42-acb1-d4f002b39edb" path="/var/lib/kubelet/pods/b308670a-b98f-4b42-acb1-d4f002b39edb/volumes"
Nov 02 15:04:35.989206 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:35.989167    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5" event={"ID":"29a0b005-68cb-408d-8375-c01427527d33","Type":"ContainerStarted","Data":"0a1ff91c474a8c2cbed7cda6f2ecedac9100f39e06c0dd34622e2e0dd46413d9"}
Nov 02 15:04:35.989447 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:35.989221    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5" event={"ID":"29a0b005-68cb-408d-8375-c01427527d33","Type":"ContainerStarted","Data":"22eb29af15fba1f0b04d6dc67019b50151947855c834e2c0f3790e33699bbf33"}
Nov 02 15:04:36.035308 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:36.035246    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5" podStartSLOduration=2.035225647 podStartE2EDuration="2.035225647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:36.031442395 +0000 UTC m=+4040.575157991" watchObservedRunningTime="2026-11-02 15:04:36.035225647 +0000 UTC m=+4040.578941244"
Nov 02 15:04:36.996308 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:36.996267    4492 generic.go:356] "Generic (PLEG): container finished" podID="29a0b005-68cb-408d-8375-c01427527d33" containerID="0a1ff91c474a8c2cbed7cda6f2ecedac9100f39e06c0dd34622e2e0dd46413d9" exitCode=0
Nov 02 15:04:36.996771 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:36.996371    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5" event={"ID":"29a0b005-68cb-408d-8375-c01427527d33","Type":"ContainerDied","Data":"0a1ff91c474a8c2cbed7cda6f2ecedac9100f39e06c0dd34622e2e0dd46413d9"}
Nov 02 15:04:38.150900 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.150871    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:38.284672 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.284592    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg\" (UniqueName: \"kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg\") pod \"29a0b005-68cb-408d-8375-c01427527d33\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") "
Nov 02 15:04:38.284838 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.284685    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca\") pod \"29a0b005-68cb-408d-8375-c01427527d33\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") "
Nov 02 15:04:38.284838 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.284734    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca\") pod \"29a0b005-68cb-408d-8375-c01427527d33\" (UID: \"29a0b005-68cb-408d-8375-c01427527d33\") "
Nov 02 15:04:38.285118 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.285093    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca" pod "29a0b005-68cb-408d-8375-c01427527d33" (UID: "29a0b005-68cb-408d-8375-c01427527d33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:04:38.286964 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.286936    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca" pod "29a0b005-68cb-408d-8375-c01427527d33" (UID: "29a0b005-68cb-408d-8375-c01427527d33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:04:38.287085 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.287066    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg" pod "29a0b005-68cb-408d-8375-c01427527d33" (UID: "29a0b005-68cb-408d-8375-c01427527d33"). InnerVolumeSpecName "kube-api-access-lv9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:04:38.386121 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.386080    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv9wg\" (UniqueName: \"kubernetes.io/projected/29a0b005-68cb-408d-8375-c01427527d33-kube-api-access-lv9wg\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:38.386121 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.386115    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a0b005-68cb-408d-8375-c01427527d33-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:38.386121 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:38.386125    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a0b005-68cb-408d-8375-c01427527d33-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:04:39.012498 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:39.012464    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"
Nov 02 15:04:39.012724 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:39.012460    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5" event={"ID":"29a0b005-68cb-408d-8375-c01427527d33","Type":"ContainerDied","Data":"22eb29af15fba1f0b04d6dc67019b50151947855c834e2c0f3790e33699bbf33"}
Nov 02 15:04:39.012724 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:39.012598    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22eb29af15fba1f0b04d6dc67019b50151947855c834e2c0f3790e33699bbf33"
Nov 02 15:04:46.516765 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:04:46.516715    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:04:49.687210 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:49.687180    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:50.108649 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.108555    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"]
Nov 02 15:04:50.109267 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.109242    4492 cpu_manager.go:414] "containerMap: removing container" podUID="29a0b005-68cb-408d-8375-c01427527d33" containerName="revoke-keys"
Nov 02 15:04:50.109423 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.109268    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a0b005-68cb-408d-8375-c01427527d33" containerName="revoke-keys"
Nov 02 15:04:50.109423 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.109403    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a0b005-68cb-408d-8375-c01427527d33" containerName="revoke-keys"
Nov 02 15:04:50.113392 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.113344    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.127916 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.127632    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4bc210-openshift-default-dockercfg-hx9m8\""
Nov 02 15:04:50.142031 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.141985    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"]
Nov 02 15:04:50.158877 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.158847    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:50.221410 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221328    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221726    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221787    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221814    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k7d\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221891    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221916    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.221985    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.222022    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.222286 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.222066    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.322883 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.322844    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.322883 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.322890    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.322907    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k7d\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.322952    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.322968    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323015    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323049    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323089    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323156 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323129    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323552 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323372    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323552 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323463    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323629 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323601    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.323665 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323639    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.324031 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.323960    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.325863 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.325831    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.326100 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.326083    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.344961 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.344931    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.345106 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.345086    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k7d\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d\") pod \"e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") " pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.430338 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.430303    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:50.539243 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.539203    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:50.752580 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.752500    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"]
Nov 02 15:04:50.758577 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:04:50.758545    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f140bb_0291_4b9d_bb48_d0a0a8a7f1ff.slice/crio-b06f1236b9f7005a680d64b800bb92878b60c47722ebcf402ffdd75cb3c4bf56 WatchSource:0}: Error finding container b06f1236b9f7005a680d64b800bb92878b60c47722ebcf402ffdd75cb3c4bf56: Status 404 returned error can't find the container with id b06f1236b9f7005a680d64b800bb92878b60c47722ebcf402ffdd75cb3c4bf56
Nov 02 15:04:50.761251 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.761220    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 15:04:50.761347 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.761290    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 15:04:50.761347 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:50.761320    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 15:04:51.087574 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:51.087483    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" event={"ID":"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff","Type":"ContainerStarted","Data":"871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"}
Nov 02 15:04:51.087574 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:51.087553    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" event={"ID":"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff","Type":"ContainerStarted","Data":"b06f1236b9f7005a680d64b800bb92878b60c47722ebcf402ffdd75cb3c4bf56"}
Nov 02 15:04:51.151345 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:51.151284    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podStartSLOduration=1.151267919 podStartE2EDuration="1.151267919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:51.148948084 +0000 UTC m=+4055.692663692" watchObservedRunningTime="2026-11-02 15:04:51.151267919 +0000 UTC m=+4055.694983513"
Nov 02 15:04:51.431314 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:51.431283    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:51.436834 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:51.436809    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:52.093416 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:52.093381    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:52.094338 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:52.094317    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:04:53.561909 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:53.561875    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:53.700876 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:53.700841    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:53.831670 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:53.831578    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:55.355322 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:55.355290    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:55.475462 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:55.475430    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:57.282015 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:57.281982    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:59.115516 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:04:59.115473    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:00.913300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:00.913263    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:02.828318 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:02.828276    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:03.523832 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:03.523800    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:04.714147 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:04.714110    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:06.685243 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:06.685201    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:08.134537 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:08.134490    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:05:08.135061 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:08.134558    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:05:08.510343 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:08.510301    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:08.617316 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:08.617279    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:09.197508 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.197449    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:09.215797 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.215713    4492 generic.go:356] "Generic (PLEG): container finished" podID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerID="871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d" exitCode=0
Nov 02 15:05:09.215970 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.215790    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" event={"ID":"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff","Type":"ContainerDied","Data":"871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"}
Nov 02 15:05:09.216300 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.216283    4492 scope.go:122] "RemoveContainer" containerID="871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"
Nov 02 15:05:09.217755 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.217711    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 15:05:09.217898 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.217797    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 15:05:09.217898 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.217847    4492 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 15:05:09.326132 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.326093    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:09.433705 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:09.433635    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:10.229145 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:10.229113    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" event={"ID":"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff","Type":"ContainerStarted","Data":"acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07"}
Nov 02 15:05:10.410717 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:10.410680    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:10.431509 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:10.431477    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:05:10.431752 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:10.431728    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 15:05:10.431802 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:10.431783    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 15:05:11.205817 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:11.205773    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:11.431955 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:11.431917    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 15:05:11.432719 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:11.432687    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 15:05:12.349004 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:12.348902    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:12.431171 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:12.431136    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 15:05:12.431387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:12.431190    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 15:05:13.431384 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:13.431324    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 15:05:13.431806 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:13.431409    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 15:05:14.259095 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:14.259058    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:14.431194 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:14.431161    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 15:05:14.431406 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:14.431231    4492 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 15:05:15.741836 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:15.741804    4492 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:05:15.743123 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:15.743101    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:05:15.744056 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:15.744036    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:05:16.088006 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:16.087923    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:16.516618 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:05:16.516582    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:05:17.896408 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:17.896374    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:19.676498 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:19.676460    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:21.408185 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:21.408147    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:23.204114 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:23.204060    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:25.006138 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:25.006105    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:26.722073 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:26.721980    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:28.092646 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:28.092615    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:28.585999 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:28.585970    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:30.401769 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:30.401722    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:32.198373 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:32.198329    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:33.046479 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:33.046444    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:34.001818 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:34.001784    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:35.804036 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:35.804001    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:37.693927 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:37.693893    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:39.031062 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:39.031024    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"]
Nov 02 15:05:39.040628 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:39.040598    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-js5fh"]
Nov 02 15:05:39.489919 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:39.489881    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:39.579932 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:39.579897    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83732677-acea-47fa-9b17-486c2f33efc9" path="/var/lib/kubelet/pods/83732677-acea-47fa-9b17-486c2f33efc9/volumes"
Nov 02 15:05:41.293851 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:41.293820    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:43.094921 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.094888    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:43.109482 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.109449    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"]
Nov 02 15:05:43.113542 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.113523    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.132468 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.132441    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"]
Nov 02 15:05:43.138711 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.138663    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:05:43.139205 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.139186    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:05:43.139577 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.139560    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:05:43.140001 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.139982    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:05:43.269010 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.268970    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.269174 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.269088    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.269174 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.269166    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr6c\" (UniqueName: \"kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.369787 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.369696    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.369969 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.369819    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr6c\" (UniqueName: \"kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.369969 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.369859    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.370581 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.370562    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.372311 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.372279    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.391637 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.391607    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr6c\" (UniqueName: \"kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c\") pod \"maas-api-revoke-sub-29d23d6f-2l576\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.426540 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.426503    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:43.731606 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:43.731572    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"]
Nov 02 15:05:43.733009 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:05:43.732978    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b849642_93ba_4111_8394_4568dcd39432.slice/crio-2a4fe3f4733cc97e6e3b831047af6c7771ad6e2df094b2aba8ef8a8eea067c14 WatchSource:0}: Error finding container 2a4fe3f4733cc97e6e3b831047af6c7771ad6e2df094b2aba8ef8a8eea067c14: Status 404 returned error can't find the container with id 2a4fe3f4733cc97e6e3b831047af6c7771ad6e2df094b2aba8ef8a8eea067c14
Nov 02 15:05:44.465403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:44.465348    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576" event={"ID":"4b849642-93ba-4111-8394-4568dcd39432","Type":"ContainerStarted","Data":"860236ecc86a31415a8e403e4e62177df6dfd9ca1e25c792217e343a9f7d0c23"}
Nov 02 15:05:44.465785 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:44.465435    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576" event={"ID":"4b849642-93ba-4111-8394-4568dcd39432","Type":"ContainerStarted","Data":"2a4fe3f4733cc97e6e3b831047af6c7771ad6e2df094b2aba8ef8a8eea067c14"}
Nov 02 15:05:44.504499 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:44.504442    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576" podStartSLOduration=1.5044257650000001 podStartE2EDuration="1.504425765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:44.500979196 +0000 UTC m=+4109.044694785" watchObservedRunningTime="2026-11-02 15:05:44.504425765 +0000 UTC m=+4109.048141359"
Nov 02 15:05:44.889823 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:44.889790    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:45.473125 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:45.473092    4492 generic.go:356] "Generic (PLEG): container finished" podID="4b849642-93ba-4111-8394-4568dcd39432" containerID="860236ecc86a31415a8e403e4e62177df6dfd9ca1e25c792217e343a9f7d0c23" exitCode=0
Nov 02 15:05:45.473609 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:45.473183    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576" event={"ID":"4b849642-93ba-4111-8394-4568dcd39432","Type":"ContainerDied","Data":"860236ecc86a31415a8e403e4e62177df6dfd9ca1e25c792217e343a9f7d0c23"}
Nov 02 15:05:46.521055 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:05:46.521012    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:05:46.606737 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.606702    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:46.628170 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.628146    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:46.702053 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.702017    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca\") pod \"4b849642-93ba-4111-8394-4568dcd39432\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") "
Nov 02 15:05:46.702053 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.702058    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c\" (UniqueName: \"kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c\") pod \"4b849642-93ba-4111-8394-4568dcd39432\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") "
Nov 02 15:05:46.702326 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.702131    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca\") pod \"4b849642-93ba-4111-8394-4568dcd39432\" (UID: \"4b849642-93ba-4111-8394-4568dcd39432\") "
Nov 02 15:05:46.702494 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.702467    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca" pod "4b849642-93ba-4111-8394-4568dcd39432" (UID: "4b849642-93ba-4111-8394-4568dcd39432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:05:46.704710 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.704688    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c" pod "4b849642-93ba-4111-8394-4568dcd39432" (UID: "4b849642-93ba-4111-8394-4568dcd39432"). InnerVolumeSpecName "kube-api-access-hlr6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:05:46.705426 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.705405    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca" pod "4b849642-93ba-4111-8394-4568dcd39432" (UID: "4b849642-93ba-4111-8394-4568dcd39432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:05:46.803323 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.803247    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b849642-93ba-4111-8394-4568dcd39432-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:05:46.803323 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.803274    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlr6c\" (UniqueName: \"kubernetes.io/projected/4b849642-93ba-4111-8394-4568dcd39432-kube-api-access-hlr6c\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:05:46.803323 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:46.803285    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b849642-93ba-4111-8394-4568dcd39432-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:05:47.485957 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:47.485926    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"
Nov 02 15:05:47.486141 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:47.485962    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576" event={"ID":"4b849642-93ba-4111-8394-4568dcd39432","Type":"ContainerDied","Data":"2a4fe3f4733cc97e6e3b831047af6c7771ad6e2df094b2aba8ef8a8eea067c14"}
Nov 02 15:05:47.486141 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:47.486013    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4fe3f4733cc97e6e3b831047af6c7771ad6e2df094b2aba8ef8a8eea067c14"
Nov 02 15:05:48.388620 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:48.388583    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:50.190192 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:50.190157    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:51.996252 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:51.996221    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.798258 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:53.798228    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:54.164330 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:54.164258    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:54.685763 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:54.685730    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:55.606288 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:55.606250    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.499504 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:57.499468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:59.192030 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:59.191997    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:59.309202 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:05:59.309167    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:01.100245 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:01.100212    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:01.373082 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:01.372987    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:01.494610 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:01.494577    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:01.609071 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:01.609033    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:02.029426 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:02.029388    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"]
Nov 02 15:06:02.038789 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:02.038757    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rx4qc"]
Nov 02 15:06:02.905506 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:02.905468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:03.579707 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:03.579672    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1541d59-07f6-46f3-b7f0-7128d43e27b6" path="/var/lib/kubelet/pods/a1541d59-07f6-46f3-b7f0-7128d43e27b6/volumes"
Nov 02 15:06:03.655184 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:03.655148    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:04.393280 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:04.393248    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:07.575105 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:07.575068    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:07.686971 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:07.686941    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:14.033486 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:14.033450    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"]
Nov 02 15:06:14.042878 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:14.042851    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-29tkg"]
Nov 02 15:06:15.587807 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:15.587772    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f3c0a3-90ab-41b1-b95a-8cb5af424f99" path="/var/lib/kubelet/pods/72f3c0a3-90ab-41b1-b95a-8cb5af424f99/volumes"
Nov 02 15:06:16.517333 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:06:16.517295    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:06:28.225450 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:28.225417    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:33.170379 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:33.170325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:35.053525 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:35.053483    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:35.160679 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:35.160640    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:36.554786 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:36.554750    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:36.678649 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:36.678592    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:36.809261 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:36.809178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:36.937821 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:36.937783    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:38.393534 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:38.393502    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:38.720884 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:38.720850    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:40.201035 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:40.201005    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:42.000382 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:42.000325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:43.032179 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:43.032141    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"]
Nov 02 15:06:43.041087 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:43.041055    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zksbt"]
Nov 02 15:06:43.580718 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:43.580685    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70d513b-a32b-4154-9d13-fcc961495719" path="/var/lib/kubelet/pods/a70d513b-a32b-4154-9d13-fcc961495719/volumes"
Nov 02 15:06:43.794547 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:43.794512    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:44.948827 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:44.948791    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:45.060045 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.059998    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:45.347848 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.347747    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"]
Nov 02 15:06:45.348815 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.348786    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4b849642-93ba-4111-8394-4568dcd39432" containerName="revoke-keys"
Nov 02 15:06:45.348980 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.348967    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b849642-93ba-4111-8394-4568dcd39432" containerName="revoke-keys"
Nov 02 15:06:45.349207 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.349196    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b849642-93ba-4111-8394-4568dcd39432" containerName="revoke-keys"
Nov 02 15:06:45.353452 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.353426    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.369060 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.369027    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.369839 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.369814    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4p6\" (UniqueName: \"kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.370294 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.370183    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.382859 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.380663    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:06:45.382859 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.380789    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:06:45.382859 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.380851    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:06:45.382859 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.380907    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:06:45.385240 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.385201    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"]
Nov 02 15:06:45.471394 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.471345    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.471594 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.471446    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.471594 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.471497    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4p6\" (UniqueName: \"kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.471986 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.471965    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.473938 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.473914    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.495525 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.495499    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4p6\" (UniqueName: \"kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:45.679603 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.679553    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:45.685498 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:45.685473    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:46.002563 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:46.002447    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"]
Nov 02 15:06:46.005518 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:06:46.005484    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04765db_1449_4446_9ce2_38c81139c755.slice/crio-40e96f2b2623056081453f39782cfff51232655c13202af9d5c8b1d6e8b4fc59 WatchSource:0}: Error finding container 40e96f2b2623056081453f39782cfff51232655c13202af9d5c8b1d6e8b4fc59: Status 404 returned error can't find the container with id 40e96f2b2623056081453f39782cfff51232655c13202af9d5c8b1d6e8b4fc59
Nov 02 15:06:46.516628 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:06:46.516594    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:06:46.880559 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:46.880516    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7" event={"ID":"e04765db-1449-4446-9ce2-38c81139c755","Type":"ContainerStarted","Data":"acfe4358d7013928e5dfdbb8764e3b99f755249fb068e17b53dc5589998569fc"}
Nov 02 15:06:46.880769 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:46.880578    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7" event={"ID":"e04765db-1449-4446-9ce2-38c81139c755","Type":"ContainerStarted","Data":"40e96f2b2623056081453f39782cfff51232655c13202af9d5c8b1d6e8b4fc59"}
Nov 02 15:06:46.919115 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:46.919060    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7" podStartSLOduration=1.919046533 podStartE2EDuration="1.919046533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:46.916322577 +0000 UTC m=+4171.460038173" watchObservedRunningTime="2026-11-02 15:06:46.919046533 +0000 UTC m=+4171.462762126"
Nov 02 15:06:47.493982 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:47.493951    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:47.887982 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:47.887895    4492 generic.go:356] "Generic (PLEG): container finished" podID="e04765db-1449-4446-9ce2-38c81139c755" containerID="acfe4358d7013928e5dfdbb8764e3b99f755249fb068e17b53dc5589998569fc" exitCode=0
Nov 02 15:06:47.888149 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:47.887982    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7" event={"ID":"e04765db-1449-4446-9ce2-38c81139c755","Type":"ContainerDied","Data":"acfe4358d7013928e5dfdbb8764e3b99f755249fb068e17b53dc5589998569fc"}
Nov 02 15:06:49.037900 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.037866    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:49.108837 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.108803    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca\") pod \"e04765db-1449-4446-9ce2-38c81139c755\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") "
Nov 02 15:06:49.108837 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.108845    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6\" (UniqueName: \"kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6\") pod \"e04765db-1449-4446-9ce2-38c81139c755\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") "
Nov 02 15:06:49.109145 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.108899    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca\") pod \"e04765db-1449-4446-9ce2-38c81139c755\" (UID: \"e04765db-1449-4446-9ce2-38c81139c755\") "
Nov 02 15:06:49.109326 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.109298    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca" pod "e04765db-1449-4446-9ce2-38c81139c755" (UID: "e04765db-1449-4446-9ce2-38c81139c755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:06:49.111302 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.111271    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca" pod "e04765db-1449-4446-9ce2-38c81139c755" (UID: "e04765db-1449-4446-9ce2-38c81139c755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:06:49.111302 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.111279    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6" pod "e04765db-1449-4446-9ce2-38c81139c755" (UID: "e04765db-1449-4446-9ce2-38c81139c755"). InnerVolumeSpecName "kube-api-access-tj4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:06:49.210375 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.210330    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04765db-1449-4446-9ce2-38c81139c755-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:06:49.210375 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.210374    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4p6\" (UniqueName: \"kubernetes.io/projected/e04765db-1449-4446-9ce2-38c81139c755-kube-api-access-tj4p6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:06:49.210585 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.210386    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04765db-1449-4446-9ce2-38c81139c755-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:06:49.221787 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.221755    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:49.901799 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.901701    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"
Nov 02 15:06:49.901799 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.901719    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7" event={"ID":"e04765db-1449-4446-9ce2-38c81139c755","Type":"ContainerDied","Data":"40e96f2b2623056081453f39782cfff51232655c13202af9d5c8b1d6e8b4fc59"}
Nov 02 15:06:49.901799 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:49.901776    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e96f2b2623056081453f39782cfff51232655c13202af9d5c8b1d6e8b4fc59"
Nov 02 15:06:51.012369 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:51.012335    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:52.819524 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:52.819487    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:53.032523 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:53.032492    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"]
Nov 02 15:06:53.045046 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:53.045011    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v29xf"]
Nov 02 15:06:53.581256 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:53.581216    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1a993f-acae-4917-881d-80ce69f09a0c" path="/var/lib/kubelet/pods/bc1a993f-acae-4917-881d-80ce69f09a0c/volumes"
Nov 02 15:06:54.587144 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:54.587112    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:55.981634 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:55.981605    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:56.099602 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:56.099568    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:56.392079 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:56.391979    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:57.863042 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:57.863012    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:58.393195 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.393148    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:58.503092 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.503063    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:58.574418 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.574378    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"]
Nov 02 15:06:58.583930 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.583892    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4bc210-pv6r7"]
Nov 02 15:06:58.905755 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.905716    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"]
Nov 02 15:06:58.906253 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.906016    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" containerID="cri-o://acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07" gracePeriod=30
Nov 02 15:06:58.949519 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:58.949428    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:59.580870 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:06:59.580832    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04765db-1449-4446-9ce2-38c81139c755" path="/var/lib/kubelet/pods/e04765db-1449-4446-9ce2-38c81139c755/volumes"
Nov 02 15:07:00.395088 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.395051    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"]
Nov 02 15:07:00.404853 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.404825    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cpzb5"]
Nov 02 15:07:00.447187 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.447156    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"]
Nov 02 15:07:00.447627 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.447613    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e04765db-1449-4446-9ce2-38c81139c755" containerName="revoke-keys"
Nov 02 15:07:00.447670 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.447629    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04765db-1449-4446-9ce2-38c81139c755" containerName="revoke-keys"
Nov 02 15:07:00.447735 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.447725    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04765db-1449-4446-9ce2-38c81139c755" containerName="revoke-keys"
Nov 02 15:07:00.451213 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.451192    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.465817 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.465787    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:07:00.466039 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.466009    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:07:00.466039 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.465804    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:07:00.466239 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.466133    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:07:00.471309 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.471274    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"]
Nov 02 15:07:00.555341 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.555297    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.555568 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.555448    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67hm\" (UniqueName: \"kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.555568 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.555542    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.657045 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.656950    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67hm\" (UniqueName: \"kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.657045 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.657028    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.657317 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.657156    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.657883 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.657843    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.659685 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.659663    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.680468 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.680439    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67hm\" (UniqueName: \"kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm\") pod \"maas-api-revoke-sub-3d4156f1-nh27h\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:00.745863 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.745817    4492 patch_prober.go:29] interesting pod/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:07:00.746044 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.745903    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:07:00.764492 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:00.764460    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:01.068584 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:01.068542    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"]
Nov 02 15:07:01.072062 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:07:01.072027    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc01afc_de5f_4ba2_bab0_d48a74cb497b.slice/crio-76ba05eaf6ee3f207f9fa17157fc6b6f2242a4ddcd54d3e4cdae9142f3d67e78 WatchSource:0}: Error finding container 76ba05eaf6ee3f207f9fa17157fc6b6f2242a4ddcd54d3e4cdae9142f3d67e78: Status 404 returned error can't find the container with id 76ba05eaf6ee3f207f9fa17157fc6b6f2242a4ddcd54d3e4cdae9142f3d67e78
Nov 02 15:07:01.580492 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:01.580459    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a0b005-68cb-408d-8375-c01427527d33" path="/var/lib/kubelet/pods/29a0b005-68cb-408d-8375-c01427527d33/volumes"
Nov 02 15:07:01.977294 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:01.977253    4492 generic.go:356] "Generic (PLEG): container finished" podID="4cc01afc-de5f-4ba2-bab0-d48a74cb497b" containerID="0f96495f56b775f8e6b7c79090e2d02363e12a5c592e1222118641a6a51a0c8d" exitCode=0
Nov 02 15:07:01.977532 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:01.977342    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h" event={"ID":"4cc01afc-de5f-4ba2-bab0-d48a74cb497b","Type":"ContainerDied","Data":"0f96495f56b775f8e6b7c79090e2d02363e12a5c592e1222118641a6a51a0c8d"}
Nov 02 15:07:01.977532 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:01.977414    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h" event={"ID":"4cc01afc-de5f-4ba2-bab0-d48a74cb497b","Type":"ContainerStarted","Data":"76ba05eaf6ee3f207f9fa17157fc6b6f2242a4ddcd54d3e4cdae9142f3d67e78"}
Nov 02 15:07:03.132656 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.132629    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:03.286491 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.286403    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm\" (UniqueName: \"kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm\") pod \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") "
Nov 02 15:07:03.286491 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.286491    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca\") pod \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") "
Nov 02 15:07:03.286738 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.286541    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca\") pod \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\" (UID: \"4cc01afc-de5f-4ba2-bab0-d48a74cb497b\") "
Nov 02 15:07:03.286957 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.286936    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca" pod "4cc01afc-de5f-4ba2-bab0-d48a74cb497b" (UID: "4cc01afc-de5f-4ba2-bab0-d48a74cb497b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:07:03.288762 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.288729    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca" pod "4cc01afc-de5f-4ba2-bab0-d48a74cb497b" (UID: "4cc01afc-de5f-4ba2-bab0-d48a74cb497b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:07:03.288874 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.288801    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm" pod "4cc01afc-de5f-4ba2-bab0-d48a74cb497b" (UID: "4cc01afc-de5f-4ba2-bab0-d48a74cb497b"). InnerVolumeSpecName "kube-api-access-j67hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:07:03.388450 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.388414    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67hm\" (UniqueName: \"kubernetes.io/projected/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-kube-api-access-j67hm\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:03.388450 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.388443    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:03.388450 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.388453    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc01afc-de5f-4ba2-bab0-d48a74cb497b-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:03.780502 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.780468    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:03.991653 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.991625    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"
Nov 02 15:07:03.991829 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.991675    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h" event={"ID":"4cc01afc-de5f-4ba2-bab0-d48a74cb497b","Type":"ContainerDied","Data":"76ba05eaf6ee3f207f9fa17157fc6b6f2242a4ddcd54d3e4cdae9142f3d67e78"}
Nov 02 15:07:03.991829 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:03.991724    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ba05eaf6ee3f207f9fa17157fc6b6f2242a4ddcd54d3e4cdae9142f3d67e78"
Nov 02 15:07:04.164196 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.164172    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:07:04.296472 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296428    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296472 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296475    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296752 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296506    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296752 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296588    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296752 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296613    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296752 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296714    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296773    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296847    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296826    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:07:04.296977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296896    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token\") pod \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\" (UID: \"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff\") "
Nov 02 15:07:04.296977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296899    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:07:04.296977 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.296909    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:07:04.297308 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297079    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:07:04.297308 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297179    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:07:04.297449 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297332    4492 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.297449 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297390    4492 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-credential-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.297449 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297405    4492 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-workload-socket\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.297449 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297420    4492 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-data\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.297449 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.297435    4492 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istiod-ca-cert\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.299168 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.299132    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "kube-api-access-d9k7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:07:04.299168 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.299150    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:07:04.299633 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.299614    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:07:04.299699 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.299626    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo" pod "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" (UID: "32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:07:04.398413 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.398371    4492 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-envoy\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.398413 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.398405    4492 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-token\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.398413 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.398414    4492 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-istio-podinfo\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:04.398662 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:04.398424    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9k7d\" (UniqueName: \"kubernetes.io/projected/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff-kube-api-access-d9k7d\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:05.001185 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.001149    4492 generic.go:356] "Generic (PLEG): container finished" podID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerID="acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07" exitCode=0
Nov 02 15:07:05.001417 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.001202    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" event={"ID":"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff","Type":"ContainerDied","Data":"acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07"}
Nov 02 15:07:05.001417 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.001228    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"
Nov 02 15:07:05.001417 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.001246    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x" event={"ID":"32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff","Type":"ContainerDied","Data":"b06f1236b9f7005a680d64b800bb92878b60c47722ebcf402ffdd75cb3c4bf56"}
Nov 02 15:07:05.001417 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.001267    4492 scope.go:122] "RemoveContainer" containerID="acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07"
Nov 02 15:07:05.011941 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.011922    4492 scope.go:122] "RemoveContainer" containerID="871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"
Nov 02 15:07:05.021345 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.021180    4492 scope.go:122] "RemoveContainer" containerID="acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07"
Nov 02 15:07:05.021517 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:07:05.021492    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07\": container with ID starting with acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07 not found: ID does not exist" containerID="acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07"
Nov 02 15:07:05.021595 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.021532    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07"} err="failed to get container status \"acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07\": rpc error: code = NotFound desc = could not find container \"acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07\": container with ID starting with acdc51f60805856fab5322e8893c3a41428ec2824ec6e9889aaa9d62b973ee07 not found: ID does not exist"
Nov 02 15:07:05.021595 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.021561    4492 scope.go:122] "RemoveContainer" containerID="871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"
Nov 02 15:07:05.021841 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:07:05.021822    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d\": container with ID starting with 871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d not found: ID does not exist" containerID="871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"
Nov 02 15:07:05.021903 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.021849    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d"} err="failed to get container status \"871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d\": rpc error: code = NotFound desc = could not find container \"871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d\": container with ID starting with 871b33f607afad4cac411d262c298e281a032d99bc372504dde787cf1bc00c1d not found: ID does not exist"
Nov 02 15:07:05.058199 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.058162    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"]
Nov 02 15:07:05.068602 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.068570    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4bc210-openshift-default-8689fcffdfv8k9x"]
Nov 02 15:07:05.580494 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:05.580462    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" path="/var/lib/kubelet/pods/32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff/volumes"
Nov 02 15:07:14.031174 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:14.031135    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"]
Nov 02 15:07:14.041208 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:14.041180    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cp8q4"]
Nov 02 15:07:15.580194 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:15.580158    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cfb406-4ece-42a7-9d05-26b862f9c3ac" path="/var/lib/kubelet/pods/37cfb406-4ece-42a7-9d05-26b862f9c3ac/volumes"
Nov 02 15:07:16.516626 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:07:16.516589    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:07:18.026252 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:18.026207    4492 scope.go:122] "RemoveContainer" containerID="ebe615d855732ac28587cb57da3575a7bc613e0593583f5d80ce3de28ef0779a"
Nov 02 15:07:18.042738 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:18.042506    4492 scope.go:122] "RemoveContainer" containerID="7d76ff7ebc6812ff04c59f1f3e96ee34aa3827933aad1edc7ef35da66ec10df9"
Nov 02 15:07:18.059461 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:18.059433    4492 scope.go:122] "RemoveContainer" containerID="b461b7424364f60aa5e2bbe882faa2eecbdca3c14b643febb48504defb36ca3c"
Nov 02 15:07:19.954555 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:19.954502    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zdbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:07:19.955738 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:19.955705    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zdbqj" podUID="6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:07:19.974483 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:19.974449    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-zvq6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:07:19.974729 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:19.974702    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-zvq6k" podUID="cfaf084d-3acb-4659-8503-22bcfc06ee89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:07:20.042933 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:20.042454    4492 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-89nq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:07:20.042933 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:20.042537    4492 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-89nq9" podUID="15d4191a-d188-4436-b784-b5e376fc5ef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:07:21.107056 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107017    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"]
Nov 02 15:07:21.107503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107441    4492 cpu_manager.go:414] "containerMap: removing container" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy"
Nov 02 15:07:21.107503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107453    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy"
Nov 02 15:07:21.107503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107466    4492 cpu_manager.go:414] "containerMap: removing container" podUID="4cc01afc-de5f-4ba2-bab0-d48a74cb497b" containerName="revoke-keys"
Nov 02 15:07:21.107503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107471    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc01afc-de5f-4ba2-bab0-d48a74cb497b" containerName="revoke-keys"
Nov 02 15:07:21.107664 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107551    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy"
Nov 02 15:07:21.107664 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107562    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy"
Nov 02 15:07:21.107664 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.107569    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc01afc-de5f-4ba2-bab0-d48a74cb497b" containerName="revoke-keys"
Nov 02 15:07:21.112952 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.112926    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.130844 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.130818    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"]
Nov 02 15:07:21.136458 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.136426    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:07:21.136678 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.136438    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\""
Nov 02 15:07:21.136678 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.136493    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:07:21.136678 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.136532    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:07:21.240926 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.240887    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.241135 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.240968    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zh6\" (UniqueName: \"kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.241188 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.241160    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.341851 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.341814    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.342038 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.341915    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.342038 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.341985    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zh6\" (UniqueName: \"kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.342560 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.342533    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.344502 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.344475    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.364031 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.363968    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zh6\" (UniqueName: \"kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6\") pod \"maas-api-revoke-sub-d2bc93bc-86q92\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.428511 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.428478    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:21.731214 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:21.731182    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"]
Nov 02 15:07:21.734871 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:07:21.734840    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ffe460_5008_4a5d_a437_3cd8226a2cd7.slice/crio-8d85bdc8a1a4d0a76e23f570534abdc2a500d481e5572004a79483df4731f339 WatchSource:0}: Error finding container 8d85bdc8a1a4d0a76e23f570534abdc2a500d481e5572004a79483df4731f339: Status 404 returned error can't find the container with id 8d85bdc8a1a4d0a76e23f570534abdc2a500d481e5572004a79483df4731f339
Nov 02 15:07:22.195702 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:22.195665    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92" event={"ID":"e3ffe460-5008-4a5d-a437-3cd8226a2cd7","Type":"ContainerStarted","Data":"9827902674e834ea6a727cbd294dc28b7d70634ed4bfad020b18f82e4dbbe180"}
Nov 02 15:07:22.196126 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:22.195722    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92" event={"ID":"e3ffe460-5008-4a5d-a437-3cd8226a2cd7","Type":"ContainerStarted","Data":"8d85bdc8a1a4d0a76e23f570534abdc2a500d481e5572004a79483df4731f339"}
Nov 02 15:07:22.237552 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:22.237499    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92" podStartSLOduration=1.237482709 podStartE2EDuration="1.237482709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:22.235160246 +0000 UTC m=+4206.778875857" watchObservedRunningTime="2026-11-02 15:07:22.237482709 +0000 UTC m=+4206.781198302"
Nov 02 15:07:23.203061 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:23.203022    4492 generic.go:356] "Generic (PLEG): container finished" podID="e3ffe460-5008-4a5d-a437-3cd8226a2cd7" containerID="9827902674e834ea6a727cbd294dc28b7d70634ed4bfad020b18f82e4dbbe180" exitCode=0
Nov 02 15:07:23.203534 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:23.203107    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92" event={"ID":"e3ffe460-5008-4a5d-a437-3cd8226a2cd7","Type":"ContainerDied","Data":"9827902674e834ea6a727cbd294dc28b7d70634ed4bfad020b18f82e4dbbe180"}
Nov 02 15:07:24.344279 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.344253    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:24.474973 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.474887    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca\") pod \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") "
Nov 02 15:07:24.475136 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.475032    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6\" (UniqueName: \"kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6\") pod \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") "
Nov 02 15:07:24.475136 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.475124    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca\") pod \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\" (UID: \"e3ffe460-5008-4a5d-a437-3cd8226a2cd7\") "
Nov 02 15:07:24.475522 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.475492    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca" pod "e3ffe460-5008-4a5d-a437-3cd8226a2cd7" (UID: "e3ffe460-5008-4a5d-a437-3cd8226a2cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:07:24.477234 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.477213    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca" pod "e3ffe460-5008-4a5d-a437-3cd8226a2cd7" (UID: "e3ffe460-5008-4a5d-a437-3cd8226a2cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:07:24.477234 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.477220    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6" pod "e3ffe460-5008-4a5d-a437-3cd8226a2cd7" (UID: "e3ffe460-5008-4a5d-a437-3cd8226a2cd7"). InnerVolumeSpecName "kube-api-access-x9zh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:07:24.576780 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.576745    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zh6\" (UniqueName: \"kubernetes.io/projected/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-kube-api-access-x9zh6\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:24.576780 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.576774    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-openshift-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:24.576780 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:24.576784    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ffe460-5008-4a5d-a437-3cd8226a2cd7-maas-api-xks-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:25.217033 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:25.216999    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92" event={"ID":"e3ffe460-5008-4a5d-a437-3cd8226a2cd7","Type":"ContainerDied","Data":"8d85bdc8a1a4d0a76e23f570534abdc2a500d481e5572004a79483df4731f339"}
Nov 02 15:07:25.217033 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:25.217024    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"
Nov 02 15:07:25.217284 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:25.217057    4492 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d85bdc8a1a4d0a76e23f570534abdc2a500d481e5572004a79483df4731f339"
Nov 02 15:07:26.161550 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:26.161515    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:26.269214 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:26.269178    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:28.354409 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:28.354346    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:32.030383 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:32.030334    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"]
Nov 02 15:07:32.041101 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:32.041067    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-bmgpk"]
Nov 02 15:07:33.281084 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.281050    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:33.555728 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.555648    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"]
Nov 02 15:07:33.556343 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.556316    4492 cpu_manager.go:414] "containerMap: removing container" podUID="e3ffe460-5008-4a5d-a437-3cd8226a2cd7" containerName="revoke-keys"
Nov 02 15:07:33.556343 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.556336    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ffe460-5008-4a5d-a437-3cd8226a2cd7" containerName="revoke-keys"
Nov 02 15:07:33.556503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.556376    4492 cpu_manager.go:414] "containerMap: removing container" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy"
Nov 02 15:07:33.556503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.556382    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f140bb-0291-4b9d-bb48-d0a0a8a7f1ff" containerName="istio-proxy"
Nov 02 15:07:33.556503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.556466    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ffe460-5008-4a5d-a437-3cd8226a2cd7" containerName="revoke-keys"
Nov 02 15:07:33.559827 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.559807    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.581134 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.581105    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966cd667-33eb-449b-bfd6-d26f334996ab" path="/var/lib/kubelet/pods/966cd667-33eb-449b-bfd6-d26f334996ab/volumes"
Nov 02 15:07:33.587072 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.587052    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:07:33.587199 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.587174    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-245cc\""
Nov 02 15:07:33.587199 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.587186    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:07:33.587593 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.587574    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 15:07:33.587698 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.587628    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 15:07:33.592943 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.592921    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"]
Nov 02 15:07:33.678871 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.678838    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97t8\" (UniqueName: \"kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.679025 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.678912    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.679025 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.678950    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.679154 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.679131    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.780416 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.780381    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97t8\" (UniqueName: \"kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.780599 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.780488    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.780599 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.780543    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.780675 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.780626    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.781426 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.781315    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.783277 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.783252    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.783384 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.783256    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.802223 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.802190    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97t8\" (UniqueName: \"kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8\") pod \"maas-api-55b5486c4c-4s7rl\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") " pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:33.873593 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:33.873506    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:34.182860 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:34.182830    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"]
Nov 02 15:07:34.186038 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:07:34.186009    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c98ef9_2022_4f36_b6a0_949e329818e6.slice/crio-b5d1c4b9018573943d61f972295f14317aecd6c66d28cfb99b771eccd691e714 WatchSource:0}: Error finding container b5d1c4b9018573943d61f972295f14317aecd6c66d28cfb99b771eccd691e714: Status 404 returned error can't find the container with id b5d1c4b9018573943d61f972295f14317aecd6c66d28cfb99b771eccd691e714
Nov 02 15:07:34.187459 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:34.187440    4492 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:07:34.272867 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:34.272826    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl" event={"ID":"c0c98ef9-2022-4f36-b6a0-949e329818e6","Type":"ContainerStarted","Data":"b5d1c4b9018573943d61f972295f14317aecd6c66d28cfb99b771eccd691e714"}
Nov 02 15:07:35.280814 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:35.280770    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl" event={"ID":"c0c98ef9-2022-4f36-b6a0-949e329818e6","Type":"ContainerStarted","Data":"306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37"}
Nov 02 15:07:35.281254 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:35.280848    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:35.319203 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:35.319146    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl" podStartSLOduration=1.899116786 podStartE2EDuration="2.319126559s" totalImagesPullingTime="420.009773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:35.318255037 +0000 UTC m=+4219.861970630" watchObservedRunningTime="2026-11-02 15:07:35.319126559 +0000 UTC m=+4219.862842155"
Nov 02 15:07:41.293085 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.293049    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:41.680186 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.680156    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"]
Nov 02 15:07:41.683656 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.683639    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.703967 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.703935    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"]
Nov 02 15:07:41.760141 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.760108    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a29cfb-6653-4627-b502-b72c54d75007-maas-api-tls\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.760141 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.760150    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a29cfb-6653-4627-b502-b72c54d75007-metrics-certs\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.760387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.760259    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a29cfb-6653-4627-b502-b72c54d75007-openshift-service-ca\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.760387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.760285    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8fx\" (UniqueName: \"kubernetes.io/projected/66a29cfb-6653-4627-b502-b72c54d75007-kube-api-access-zc8fx\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.860970 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.860940    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a29cfb-6653-4627-b502-b72c54d75007-openshift-service-ca\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.860970 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.860974    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8fx\" (UniqueName: \"kubernetes.io/projected/66a29cfb-6653-4627-b502-b72c54d75007-kube-api-access-zc8fx\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.861245 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.861057    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a29cfb-6653-4627-b502-b72c54d75007-maas-api-tls\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.861245 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.861182    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a29cfb-6653-4627-b502-b72c54d75007-metrics-certs\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.861718 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.861690    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a29cfb-6653-4627-b502-b72c54d75007-openshift-service-ca\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.863669 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.863643    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a29cfb-6653-4627-b502-b72c54d75007-maas-api-tls\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.863792 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.863772    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a29cfb-6653-4627-b502-b72c54d75007-metrics-certs\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.882759 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.882734    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8fx\" (UniqueName: \"kubernetes.io/projected/66a29cfb-6653-4627-b502-b72c54d75007-kube-api-access-zc8fx\") pod \"maas-api-5956f6dcff-4kfjg\" (UID: \"66a29cfb-6653-4627-b502-b72c54d75007\") " pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:41.997198 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:41.997110    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:42.303612 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:42.303520    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"]
Nov 02 15:07:42.306541 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:07:42.306512    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a29cfb_6653_4627_b502_b72c54d75007.slice/crio-a516551a7cf08cbc445e9b07b350f0e73bea43f3852bdcbd34419e239678078f WatchSource:0}: Error finding container a516551a7cf08cbc445e9b07b350f0e73bea43f3852bdcbd34419e239678078f: Status 404 returned error can't find the container with id a516551a7cf08cbc445e9b07b350f0e73bea43f3852bdcbd34419e239678078f
Nov 02 15:07:42.326081 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:42.326050    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg" event={"ID":"66a29cfb-6653-4627-b502-b72c54d75007","Type":"ContainerStarted","Data":"a516551a7cf08cbc445e9b07b350f0e73bea43f3852bdcbd34419e239678078f"}
Nov 02 15:07:42.969873 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:42.969788    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:43.333923 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:43.333829    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg" event={"ID":"66a29cfb-6653-4627-b502-b72c54d75007","Type":"ContainerStarted","Data":"28397fab62104ef31470c5ff1ff9620ef3a9e2ace4323ec7d903b792edf84994"}
Nov 02 15:07:43.334377 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:43.334010    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:43.374260 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:43.374203    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg" podStartSLOduration=1.971291038 podStartE2EDuration="2.374188583s" totalImagesPullingTime="402.897545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:43.373035415 +0000 UTC m=+4227.916751021" watchObservedRunningTime="2026-11-02 15:07:43.374188583 +0000 UTC m=+4227.917904216"
Nov 02 15:07:46.708686 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:07:46.708628    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:07:49.348433 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:49.348393    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5956f6dcff-4kfjg"
Nov 02 15:07:49.431607 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:49.431556    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"]
Nov 02 15:07:49.432115 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:49.432078    4492 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl" podUID="c0c98ef9-2022-4f36-b6a0-949e329818e6" containerName="maas-api" containerID="cri-o://306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37" gracePeriod=30
Nov 02 15:07:50.073035 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.072738    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:50.245428 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.234677    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls\") pod \"c0c98ef9-2022-4f36-b6a0-949e329818e6\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") "
Nov 02 15:07:50.245428 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.234827    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8\" (UniqueName: \"kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8\") pod \"c0c98ef9-2022-4f36-b6a0-949e329818e6\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") "
Nov 02 15:07:50.245428 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.234939    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca\") pod \"c0c98ef9-2022-4f36-b6a0-949e329818e6\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") "
Nov 02 15:07:50.245428 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.234978    4492 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs\") pod \"c0c98ef9-2022-4f36-b6a0-949e329818e6\" (UID: \"c0c98ef9-2022-4f36-b6a0-949e329818e6\") "
Nov 02 15:07:50.253625 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.253568    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca" pod "c0c98ef9-2022-4f36-b6a0-949e329818e6" (UID: "c0c98ef9-2022-4f36-b6a0-949e329818e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:07:50.265776 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.265717    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs" pod "c0c98ef9-2022-4f36-b6a0-949e329818e6" (UID: "c0c98ef9-2022-4f36-b6a0-949e329818e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:07:50.277093 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.277046    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls" pod "c0c98ef9-2022-4f36-b6a0-949e329818e6" (UID: "c0c98ef9-2022-4f36-b6a0-949e329818e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:07:50.288570 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.288509    4492 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8" pod "c0c98ef9-2022-4f36-b6a0-949e329818e6" (UID: "c0c98ef9-2022-4f36-b6a0-949e329818e6"). InnerVolumeSpecName "kube-api-access-z97t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:07:50.337217 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.337097    4492 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98ef9-2022-4f36-b6a0-949e329818e6-openshift-service-ca\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:50.337217 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.337141    4492 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-metrics-certs\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:50.337217 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.337160    4492 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98ef9-2022-4f36-b6a0-949e329818e6-maas-api-tls\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:50.337217 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.337174    4492 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97t8\" (UniqueName: \"kubernetes.io/projected/c0c98ef9-2022-4f36-b6a0-949e329818e6-kube-api-access-z97t8\") on node \"ip-10-0-129-115.ec2.internal\" DevicePath \"\""
Nov 02 15:07:50.467403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.463788    4492 generic.go:356] "Generic (PLEG): container finished" podID="c0c98ef9-2022-4f36-b6a0-949e329818e6" containerID="306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37" exitCode=0
Nov 02 15:07:50.467403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.464004    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl" event={"ID":"c0c98ef9-2022-4f36-b6a0-949e329818e6","Type":"ContainerDied","Data":"306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37"}
Nov 02 15:07:50.467403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.464059    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl" event={"ID":"c0c98ef9-2022-4f36-b6a0-949e329818e6","Type":"ContainerDied","Data":"b5d1c4b9018573943d61f972295f14317aecd6c66d28cfb99b771eccd691e714"}
Nov 02 15:07:50.467403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.464090    4492 scope.go:122] "RemoveContainer" containerID="306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37"
Nov 02 15:07:50.467403 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.464259    4492 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"
Nov 02 15:07:50.598896 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.598764    4492 scope.go:122] "RemoveContainer" containerID="306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37"
Nov 02 15:07:50.599917 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:07:50.599814    4492 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37\": container with ID starting with 306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37 not found: ID does not exist" containerID="306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37"
Nov 02 15:07:50.599917 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.599857    4492 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37"} err="failed to get container status \"306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37\": rpc error: code = NotFound desc = could not find container \"306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37\": container with ID starting with 306ceddbe830590bb8e3d7e285c3f88be6a6e058fb6f6ae8ec625ad98bd39f37 not found: ID does not exist"
Nov 02 15:07:50.606458 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.606419    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"]
Nov 02 15:07:50.613751 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:50.613704    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55b5486c4c-4s7rl"]
Nov 02 15:07:51.056656 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:51.056626    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"]
Nov 02 15:07:51.079390 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:51.074510    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l7ckg"]
Nov 02 15:07:51.605638 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:51.605602    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6070219e-d570-486c-92f0-443d5f11bb36" path="/var/lib/kubelet/pods/6070219e-d570-486c-92f0-443d5f11bb36/volumes"
Nov 02 15:07:51.611084 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:07:51.611051    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c98ef9-2022-4f36-b6a0-949e329818e6" path="/var/lib/kubelet/pods/c0c98ef9-2022-4f36-b6a0-949e329818e6/volumes"
Nov 02 15:08:03.918121 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:03.918065    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:05.032726 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:05.032689    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"]
Nov 02 15:08:05.042430 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:05.042401    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4pmxc"]
Nov 02 15:08:05.579558 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:05.579527    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da365474-339d-43ed-8928-f6f440947989" path="/var/lib/kubelet/pods/da365474-339d-43ed-8928-f6f440947989/volumes"
Nov 02 15:08:16.518472 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:08:16.518423    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:08:18.121624 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:18.121589    4492 scope.go:122] "RemoveContainer" containerID="87ae35e49f317e574661e9f73f8f894dfa90bf363226b851f696de30984f40f6"
Nov 02 15:08:18.132345 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:18.132274    4492 scope.go:122] "RemoveContainer" containerID="df54805da4aecc12a5b5044596d8f23461a3b3d0829bb29e32149a1970af09d9"
Nov 02 15:08:18.141888 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:18.141868    4492 scope.go:122] "RemoveContainer" containerID="9468c8abe7fcd29ab6a3b7bcd937d6d9ee6486750d525c8895028ac9d9cd4517"
Nov 02 15:08:28.456443 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:28.456325    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:33.405603 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:08:33.405564    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:46.516101 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:08:46.516060    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:09:04.023288 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:04.023244    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:05.054503 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:05.054466    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:06.102624 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:06.102586    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:11.031429 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:11.031333    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"]
Nov 02 15:09:11.041216 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:11.041189    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-blcrx"]
Nov 02 15:09:11.579781 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:11.579744    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c571ec-8c92-405f-8bde-2b78d48bd3e9" path="/var/lib/kubelet/pods/b5c571ec-8c92-405f-8bde-2b78d48bd3e9/volumes"
Nov 02 15:09:15.440685 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:15.440626    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:16.517637 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:09:16.517596    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:09:18.179369 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:18.179318    4492 scope.go:122] "RemoveContainer" containerID="30ee4114a72dffa70e0c18c3ba0e38536f4e0a11a74a2102a99fc4b70c557117"
Nov 02 15:09:18.194102 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:18.194067    4492 scope.go:122] "RemoveContainer" containerID="9989fb78b9cc393f5e81a0feabfe0be87d17d7195e3b98aa3de90b4f9715527a"
Nov 02 15:09:18.247961 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:18.247934    4492 scope.go:122] "RemoveContainer" containerID="d370593d6cd669fc071bf6052de9b252bd0a574a7dd1c03d40aa628519a58791"
Nov 02 15:09:18.263172 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:18.262822    4492 scope.go:122] "RemoveContainer" containerID="0c1e848b8f01fea8bf4b542eb3561f7c45ba1a1244762c22bc22a27a20df4e4c"
Nov 02 15:09:22.029754 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:22.029716    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"]
Nov 02 15:09:22.039734 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:22.039707    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wv6sn"]
Nov 02 15:09:23.581394 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:23.581339    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95049f11-6fbc-42db-88b1-1af202feca0d" path="/var/lib/kubelet/pods/95049f11-6fbc-42db-88b1-1af202feca0d/volumes"
Nov 02 15:09:25.193541 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:25.193492    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:25.304755 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:25.304716    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:28.586446 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:28.586407    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:33.519758 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:09:33.519723    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:46.535592 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:09:46.535523    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:10:04.148321 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:04.148280    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:16.515720 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:10:16.515679    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:10:18.319141 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:18.319106    4492 scope.go:122] "RemoveContainer" containerID="b06a8a2a244d2abbf1f18658d55031fc8d6ce52fa9e514a3117c7ad8a5607358"
Nov 02 15:10:18.328490 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:18.328469    4492 scope.go:122] "RemoveContainer" containerID="bd5e96460489969828602fdfba5bd33d95afdedd42bbffea3248906bebc3755d"
Nov 02 15:10:28.714285 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:28.714248    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:33.680763 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:33.680726    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:46.518465 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:10:46.518422    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:10:47.031341 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:47.031297    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"]
Nov 02 15:10:47.041369 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:47.041315    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d23d6f-2l576"]
Nov 02 15:10:47.579735 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:10:47.579698    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b849642-93ba-4111-8394-4568dcd39432" path="/var/lib/kubelet/pods/4b849642-93ba-4111-8394-4568dcd39432/volumes"
Nov 02 15:11:00.201339 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:00.201306    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p5lx_a1fe8ffd-d93b-42a0-a727-88d122676cbc/manager/0.log"
Nov 02 15:11:00.757437 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:00.757402    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5db8dc9fd8-zlxn4_612257d4-87df-4fde-b706-29a34b2c62c8/manager/0.log"
Nov 02 15:11:02.428202 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:02.428175    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5956f6dcff-4kfjg_66a29cfb-6653-4627-b502-b72c54d75007/maas-api/0.log"
Nov 02 15:11:03.659770 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:03.659740    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-nh27h_4cc01afc-de5f-4ba2-bab0-d48a74cb497b/revoke-keys/0.log"
Nov 02 15:11:04.297792 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:04.297752    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:16.517026 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:11:16.516984    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:11:18.364680 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:18.364638    4492 scope.go:122] "RemoveContainer" containerID="6fdfa96db77ec42c3d2e8c619a3c2a34fa4765b07ebd4437f6166d9b72fb5bf9"
Nov 02 15:11:18.373820 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:18.373798    4492 scope.go:122] "RemoveContainer" containerID="0a1ff91c474a8c2cbed7cda6f2ecedac9100f39e06c0dd34622e2e0dd46413d9"
Nov 02 15:11:25.506318 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:25.506277    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:26.759904 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:26.759825    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbpwl_c44cd0a2-9688-4f95-8590-7b095ebc93b8/manager/0.log"
Nov 02 15:11:26.973463 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:26.973431    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jfj67_28a0d1ac-075a-43b5-8319-02c228b1fa6a/manager/0.log"
Nov 02 15:11:27.468201 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:27.468162    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fgnlf_42f5db3c-64e6-4d87-ab71-01fb74715791/manager/0.log"
Nov 02 15:11:27.911130 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:27.911042    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9dxhn_dd12c716-3e42-4130-a442-dc7b3066ee12/manager/0.log"
Nov 02 15:11:28.821673 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:28.821631    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:28.933565 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:28.933527    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8_139c2d0e-6d58-405e-a9ad-fc667dcbeaaf/istio-proxy/0.log"
Nov 02 15:11:29.148756 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:29.148678    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9lb82_b76ac5b5-6284-4fa9-aac3-fb5162e09484/discovery/0.log"
Nov 02 15:11:30.201590 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:30.201559    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-4q677_9a1f3047-805a-4404-9b58-57ffe5f98e65/istio-proxy/0.log"
Nov 02 15:11:30.408165 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:30.408136    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-89nq9_15d4191a-d188-4436-b784-b5e376fc5ef1/istio-proxy/0.log"
Nov 02 15:11:31.496740 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:31.496705    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-zdbqj_6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c/istio-proxy/0.log"
Nov 02 15:11:31.719400 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:31.719337    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-zvq6k_cfaf084d-3acb-4659-8503-22bcfc06ee89/istio-proxy/0.log"
Nov 02 15:11:31.939306 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:31.939276    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr_0a412529-a227-45f2-8124-9caad6a192a9/istio-proxy/0.log"
Nov 02 15:11:33.814829 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:33.814801    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:33.994456 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:33.994424    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4zm5n_55dfb5cb-2bbe-4012-a6f0-1b4cc3c439e9/main/0.log"
Nov 02 15:11:34.392688 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:34.392654    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8454nw_b70b072d-b1a4-4d5c-8745-911dbd0aef1b/main/0.log"
Nov 02 15:11:34.788196 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:34.788164    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cf2lf_e46cbd5b-ce40-4964-a0c7-264420698f31/main/0.log"
Nov 02 15:11:45.996844 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:45.996789    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:46.115537 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:46.115503    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:46.517192 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:11:46.517153    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:11:51.154072 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:51.154029    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-csrlb_519bdf1e-bdde-4785-a71c-74c0d80ea527/global-pull-secret-syncer/0.log"
Nov 02 15:11:52.182724 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:52.182690    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log"
Nov 02 15:11:52.637648 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:52.637572    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-115.ec2.internal_aff7ff2bf177bc815214d878993baa38/haproxy/0.log"
Nov 02 15:11:59.785722 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:59.785672    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbpwl_c44cd0a2-9688-4f95-8590-7b095ebc93b8/manager/0.log"
Nov 02 15:11:59.913054 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:11:59.913024    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jfj67_28a0d1ac-075a-43b5-8319-02c228b1fa6a/manager/0.log"
Nov 02 15:12:00.247899 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:00.247860    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fgnlf_42f5db3c-64e6-4d87-ab71-01fb74715791/manager/0.log"
Nov 02 15:12:00.560198 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:00.560096    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9dxhn_dd12c716-3e42-4130-a442-dc7b3066ee12/manager/0.log"
Nov 02 15:12:01.705282 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.705244    4492 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"]
Nov 02 15:12:01.705820 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.705731    4492 cpu_manager.go:414] "containerMap: removing container" podUID="c0c98ef9-2022-4f36-b6a0-949e329818e6" containerName="maas-api"
Nov 02 15:12:01.705820 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.705745    4492 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c98ef9-2022-4f36-b6a0-949e329818e6" containerName="maas-api"
Nov 02 15:12:01.705820 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.705816    4492 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c98ef9-2022-4f36-b6a0-949e329818e6" containerName="maas-api"
Nov 02 15:12:01.708851 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.708835    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.722155 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.722120    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcxf8\"/\"openshift-service-ca.crt\""
Nov 02 15:12:01.722413 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.722142    4492 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcxf8\"/\"kube-root-ca.crt\""
Nov 02 15:12:01.735323 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.735290    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"]
Nov 02 15:12:01.766081 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.766051    4492 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcxf8\"/\"default-dockercfg-pmlxg\""
Nov 02 15:12:01.806481 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.806451    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-lib-modules\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.806647 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.806583    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-proc\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.806647 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.806612    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mm8l\" (UniqueName: \"kubernetes.io/projected/552e7592-458e-488e-9444-c5bb276e5684-kube-api-access-4mm8l\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.806722 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.806709    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-podres\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.806758 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.806732    4492 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-sys\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908290 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908253    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-proc\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908290 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908294    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mm8l\" (UniqueName: \"kubernetes.io/projected/552e7592-458e-488e-9444-c5bb276e5684-kube-api-access-4mm8l\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908348    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-podres\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908394    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-sys\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908423    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-proc\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908450    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-podres\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908486    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-lib-modules\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908498    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-sys\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.908601 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.908432    4492 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/552e7592-458e-488e-9444-c5bb276e5684-lib-modules\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:01.930974 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:01.930935    4492 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mm8l\" (UniqueName: \"kubernetes.io/projected/552e7592-458e-488e-9444-c5bb276e5684-kube-api-access-4mm8l\") pod \"perf-node-gather-daemonset-lnm4r\" (UID: \"552e7592-458e-488e-9444-c5bb276e5684\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:02.023563 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:02.023470    4492 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:02.327784 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:02.327710    4492 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"]
Nov 02 15:12:02.330968 ip-10-0-129-115 kubenswrapper[4492]: W1102 15:12:02.330929    4492 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod552e7592_458e_488e_9444_c5bb276e5684.slice/crio-f4cc90b03a979ffd45c0f9b7fa7309efebad92a886281b1ee987732761f6129c WatchSource:0}: Error finding container f4cc90b03a979ffd45c0f9b7fa7309efebad92a886281b1ee987732761f6129c: Status 404 returned error can't find the container with id f4cc90b03a979ffd45c0f9b7fa7309efebad92a886281b1ee987732761f6129c
Nov 02 15:12:02.346859 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:02.346827    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r" event={"ID":"552e7592-458e-488e-9444-c5bb276e5684","Type":"ContainerStarted","Data":"f4cc90b03a979ffd45c0f9b7fa7309efebad92a886281b1ee987732761f6129c"}
Nov 02 15:12:03.354058 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:03.354013    4492 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r" event={"ID":"552e7592-458e-488e-9444-c5bb276e5684","Type":"ContainerStarted","Data":"dd5bdb9338460f040f41aae6d492f02e714466859490045312da291c9dd2d9fb"}
Nov 02 15:12:03.354507 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:03.354094    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:03.375610 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:03.375581    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cff5b96cc-wgq5m_67121397-06f1-4c97-bc11-8d26c7a9a5c1/metrics-server/0.log"
Nov 02 15:12:03.393425 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:03.393377    4492 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r" podStartSLOduration=2.393343252 podStartE2EDuration="2.393343252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 15:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:03.390913161 +0000 UTC m=+4487.934628758" watchObservedRunningTime="2026-11-02 15:12:03.393343252 +0000 UTC m=+4487.937058845"
Nov 02 15:12:03.501871 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:03.501824    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-9r7zl_35cae0b0-182b-4ea1-bf0b-9ac313533d9b/monitoring-plugin/0.log"
Nov 02 15:12:04.033444 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:04.033410    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"]
Nov 02 15:12:04.043543 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:04.043511    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nh27h"]
Nov 02 15:12:04.412697 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:04.412613    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lj7gg_84a4265a-c047-424e-b69c-c5ca1ff31395/node-exporter/0.log"
Nov 02 15:12:04.416736 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:04.416709    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:04.532864 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:04.532839    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lj7gg_84a4265a-c047-424e-b69c-c5ca1ff31395/kube-rbac-proxy/0.log"
Nov 02 15:12:04.657640 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:04.657612    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lj7gg_84a4265a-c047-424e-b69c-c5ca1ff31395/init-textfile/0.log"
Nov 02 15:12:05.581957 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:05.581927    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc01afc-de5f-4ba2-bab0-d48a74cb497b" path="/var/lib/kubelet/pods/4cc01afc-de5f-4ba2-bab0-d48a74cb497b/volumes"
Nov 02 15:12:06.466208 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:06.466174    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-89c7555db-8rtqx_b9921d64-e034-4664-87d4-9e61f1234ac6/telemeter-client/0.log"
Nov 02 15:12:06.594886 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:06.594852    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-89c7555db-8rtqx_b9921d64-e034-4664-87d4-9e61f1234ac6/reload/0.log"
Nov 02 15:12:06.723238 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:06.723126    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-89c7555db-8rtqx_b9921d64-e034-4664-87d4-9e61f1234ac6/kube-rbac-proxy/0.log"
Nov 02 15:12:06.852891 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:06.852863    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77cc7cbd6f-78cnn_943f3fe6-e595-4e27-a5e9-79c855043ae2/thanos-query/0.log"
Nov 02 15:12:06.973947 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:06.973882    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77cc7cbd6f-78cnn_943f3fe6-e595-4e27-a5e9-79c855043ae2/kube-rbac-proxy-web/0.log"
Nov 02 15:12:07.097060 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:07.097032    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77cc7cbd6f-78cnn_943f3fe6-e595-4e27-a5e9-79c855043ae2/kube-rbac-proxy/0.log"
Nov 02 15:12:07.220453 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:07.220420    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77cc7cbd6f-78cnn_943f3fe6-e595-4e27-a5e9-79c855043ae2/prom-label-proxy/0.log"
Nov 02 15:12:07.344266 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:07.344156    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77cc7cbd6f-78cnn_943f3fe6-e595-4e27-a5e9-79c855043ae2/kube-rbac-proxy-rules/0.log"
Nov 02 15:12:07.465723 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:07.465664    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77cc7cbd6f-78cnn_943f3fe6-e595-4e27-a5e9-79c855043ae2/kube-rbac-proxy-metrics/0.log"
Nov 02 15:12:08.526370 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:08.526319    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p5lx_a1fe8ffd-d93b-42a0-a727-88d122676cbc/manager/0.log"
Nov 02 15:12:09.100183 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:09.100139    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5db8dc9fd8-zlxn4_612257d4-87df-4fde-b706-29a34b2c62c8/manager/0.log"
Nov 02 15:12:09.371857 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:09.371774    4492 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-lnm4r"
Nov 02 15:12:11.971811 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:11.971780    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p5lx_a1fe8ffd-d93b-42a0-a727-88d122676cbc/manager/0.log"
Nov 02 15:12:12.564387 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:12.564257    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5db8dc9fd8-zlxn4_612257d4-87df-4fde-b706-29a34b2c62c8/manager/0.log"
Nov 02 15:12:14.973961 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:14.973931    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bpnnm_2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b/dns/0.log"
Nov 02 15:12:15.095902 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:15.095870    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bpnnm_2a1db4ad-c5c2-4a5d-966a-641a4a3e1b0b/kube-rbac-proxy/0.log"
Nov 02 15:12:15.256203 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:15.256119    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p5lx_a1fe8ffd-d93b-42a0-a727-88d122676cbc/manager/0.log"
Nov 02 15:12:15.790606 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:15.790572    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5db8dc9fd8-zlxn4_612257d4-87df-4fde-b706-29a34b2c62c8/manager/0.log"
Nov 02 15:12:15.881978 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:15.881944    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t2wc7_3cb4aab9-6e60-4c30-bdd5-eb34ae6ceee3/dns-node-resolver/0.log"
Nov 02 15:12:15.915222 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:15.915188    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:16.522813 ip-10-0-129-115 kubenswrapper[4492]: E1102 15:12:16.522764    4492 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mkt2" containerName="extractor"
Nov 02 15:12:17.208786 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:17.208750    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fcc8bf4f4-8djlj_9251af26-a06c-4b19-a480-181565db89f9/registry/0.log"
Nov 02 15:12:17.628205 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:17.628132    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wn8xj_dd07b61c-b867-47ee-ae89-d324cbcaf032/node-ca/0.log"
Nov 02 15:12:18.401720 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:18.401685    4492 scope.go:122] "RemoveContainer" containerID="860236ecc86a31415a8e403e4e62177df6dfd9ca1e25c792217e343a9f7d0c23"
Nov 02 15:12:18.672254 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:18.672178    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8_139c2d0e-6d58-405e-a9ad-fc667dcbeaaf/istio-proxy/0.log"
Nov 02 15:12:19.360371 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:19.360321    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9lb82_b76ac5b5-6284-4fa9-aac3-fb5162e09484/discovery/0.log"
Nov 02 15:12:19.583194 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:19.583167    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8_139c2d0e-6d58-405e-a9ad-fc667dcbeaaf/istio-proxy/0.log"
Nov 02 15:12:19.680514 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:19.680484    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-4q677_9a1f3047-805a-4404-9b58-57ffe5f98e65/istio-proxy/0.log"
Nov 02 15:12:19.749661 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:19.749633    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-89nq9_15d4191a-d188-4436-b784-b5e376fc5ef1/istio-proxy/0.log"
Nov 02 15:12:20.115053 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:20.114971    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-zdbqj_6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c/istio-proxy/0.log"
Nov 02 15:12:20.179774 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:20.179730    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-zvq6k_cfaf084d-3acb-4659-8503-22bcfc06ee89/istio-proxy/0.log"
Nov 02 15:12:20.244127 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:20.244093    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr_0a412529-a227-45f2-8124-9caad6a192a9/istio-proxy/0.log"
Nov 02 15:12:20.345982 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:20.345947    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9lb82_b76ac5b5-6284-4fa9-aac3-fb5162e09484/discovery/0.log"
Nov 02 15:12:21.651270 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:21.651239    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-4q677_9a1f3047-805a-4404-9b58-57ffe5f98e65/istio-proxy/0.log"
Nov 02 15:12:21.777178 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:21.777147    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-89nq9_15d4191a-d188-4436-b784-b5e376fc5ef1/istio-proxy/0.log"
Nov 02 15:12:22.473804 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:22.473777    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-zdbqj_6fa38098-b6ca-4a0b-a5f1-7fafa2bda22c/istio-proxy/0.log"
Nov 02 15:12:22.599948 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:22.599916    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-zvq6k_cfaf084d-3acb-4659-8503-22bcfc06ee89/istio-proxy/0.log"
Nov 02 15:12:22.724427 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:22.724333    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-645b5cfbff-gb6mr_0a412529-a227-45f2-8124-9caad6a192a9/istio-proxy/0.log"
Nov 02 15:12:23.371830 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:23.371800    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbpwl_c44cd0a2-9688-4f95-8590-7b095ebc93b8/manager/0.log"
Nov 02 15:12:23.439717 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:23.439685    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jfj67_28a0d1ac-075a-43b5-8319-02c228b1fa6a/manager/0.log"
Nov 02 15:12:23.641407 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:23.641297    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fgnlf_42f5db3c-64e6-4d87-ab71-01fb74715791/manager/0.log"
Nov 02 15:12:23.791005 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:23.790967    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9dxhn_dd12c716-3e42-4130-a442-dc7b3066ee12/manager/0.log"
Nov 02 15:12:25.029747 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:25.029714    4492 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"]
Nov 02 15:12:25.040249 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:25.040221    4492 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-86q92"]
Nov 02 15:12:25.557839 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:25.557807    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8r8vr_de9d0ea2-cd91-46df-b5fb-f322312c3390/serve-healthcheck-canary/0.log"
Nov 02 15:12:25.579222 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:25.579189    4492 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ffe460-5008-4a5d-a437-3cd8226a2cd7" path="/var/lib/kubelet/pods/e3ffe460-5008-4a5d-a437-3cd8226a2cd7/volumes"
Nov 02 15:12:26.918304 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:26.918272    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mkt2_a62f730c-7d26-4b82-a8cc-4607533c8a5e/kube-rbac-proxy/0.log"
Nov 02 15:12:27.040581 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:27.040555    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mkt2_a62f730c-7d26-4b82-a8cc-4607533c8a5e/exporter/0.log"
Nov 02 15:12:27.167074 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:27.167045    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mkt2_a62f730c-7d26-4b82-a8cc-4607533c8a5e/extractor/0.log"
Nov 02 15:12:28.944814 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:28.944777    4492 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2sbs7_f9047a53-882d-4708-945a-663dcdc7dc9a/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:32.078255 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:32.078219    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-fxs5r_59380b0c-d328-4fa4-b17d-ec9d3ea382fd/manager/0.log"
Nov 02 15:12:32.235507 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:32.235481    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-fxs5r_59380b0c-d328-4fa4-b17d-ec9d3ea382fd/copy-manifests/0.log"
Nov 02 15:12:34.030689 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:34.030657    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-lvdx4_3b1ea6aa-3e78-43fa-897e-e6498cf8c676/cert-manager-cainjector/0.log"
Nov 02 15:12:34.948028 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:34.947995    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-bb66448f7-4p8z8_139c2d0e-6d58-405e-a9ad-fc667dcbeaaf/istio-proxy/0.log"
Nov 02 15:12:35.496573 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:35.496539    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-fxs5r_59380b0c-d328-4fa4-b17d-ec9d3ea382fd/manager/0.log"
Nov 02 15:12:35.544800 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:35.544771    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-fxs5r_59380b0c-d328-4fa4-b17d-ec9d3ea382fd/copy-manifests/0.log"
Nov 02 15:12:35.698398 ip-10-0-129-115 kubenswrapper[4492]: I1102 15:12:35.698364    4492 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9lb82_b76ac5b5-6284-4fa9-aac3-fb5162e09484/discovery/0.log"